[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:27:59 dist-test-slave-9cj5 krb5kdc[20855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:27:59 dist-test-slave-9cj5 krb5kdc[20855](info): set up 2 sockets
Sep 29 10:27:59 dist-test-slave-9cj5 krb5kdc[20855](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 10:28:00.387843 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:28:00 dist-test-slave-9cj5 krb5kdc[20855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:28:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:28:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
I20260929 10:28:00.526397 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:40327
--webserver_interface=127.20.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:42707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:40327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:00.645679 20868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:00.646025 20868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:00.646071 20868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:00.650637 20868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:28:00.650719 20868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:00.650741 20868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:00.650759 20868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:28:00.650775 20868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:28:00.656513 20868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:42707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:40327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:40327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:00.657594 20868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:00.658823 20868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:00.669283 20874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:00.669505 20873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:00.669283 20876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:00.669602 20868 server_base.cc:1034] running on GCE node
I20260929 10:28:00.670567 20868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:00.671933 20868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:00.673146 20868 hybrid_clock.cc:648] HybridClock initialized: now 1790677680673130 us; error 27 us; skew 500 ppm
Sep 29 10:28:00 dist-test-slave-9cj5 krb5kdc[20855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677680, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:00.676221 20868 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:28:00.677186 20868 webserver.cc:460] Webserver started at http://127.20.92.62:39255/ using document root <none> and password file <none>
I20260929 10:28:00.677845 20868 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:00.677947 20868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:00.678171 20868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:00.680315 20868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "2d6f3e811713410ca11d0720982b8a20"
format_stamp: "Formatted at 2026-09-29 10:28:00 on dist-test-slave-9cj5"
server_key: "b6e6c99fa2551fcba616a34ae4673ac9"
server_key_iv: "3d5841423eded625839c344161b6200a"
server_key_version: "encryptionkey@0"
I20260929 10:28:00.680929 20868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "2d6f3e811713410ca11d0720982b8a20"
format_stamp: "Formatted at 2026-09-29 10:28:00 on dist-test-slave-9cj5"
server_key: "b6e6c99fa2551fcba616a34ae4673ac9"
server_key_iv: "3d5841423eded625839c344161b6200a"
server_key_version: "encryptionkey@0"
I20260929 10:28:00.684916 20868 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260929 10:28:00.687654 20883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:00.689379 20868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:28:00.689563 20868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "2d6f3e811713410ca11d0720982b8a20"
format_stamp: "Formatted at 2026-09-29 10:28:00 on dist-test-slave-9cj5"
server_key: "b6e6c99fa2551fcba616a34ae4673ac9"
server_key_iv: "3d5841423eded625839c344161b6200a"
server_key_version: "encryptionkey@0"
I20260929 10:28:00.689709 20868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:00.716298 20868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:00.719616 20868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:00.719848 20868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:00.727623 20868 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:40327
I20260929 10:28:00.727634 20935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:40327 every 8 connection(s)
I20260929 10:28:00.728708 20868 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:28:00.731803 20936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:00.734619 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 20868
I20260929 10:28:00.734795 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:28:00.736150 20848 external_mini_cluster.cc:1428] Setting key 9ccce3b5887f35e18c3c8960ce4d10e3
I20260929 10:28:00.739291 20936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d6f3e811713410ca11d0720982b8a20: Bootstrap starting.
I20260929 10:28:00.742291 20936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d6f3e811713410ca11d0720982b8a20: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:00.743145 20936 log.cc:824] T 00000000000000000000000000000000 P 2d6f3e811713410ca11d0720982b8a20: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:00.745416 20936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d6f3e811713410ca11d0720982b8a20: No bootstrap required, opened a new log
I20260929 10:28:00.749832 20936 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d6f3e811713410ca11d0720982b8a20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d6f3e811713410ca11d0720982b8a20" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 40327 } }
I20260929 10:28:00.750126 20936 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d6f3e811713410ca11d0720982b8a20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:00.750190 20936 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d6f3e811713410ca11d0720982b8a20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d6f3e811713410ca11d0720982b8a20, State: Initialized, Role: FOLLOWER
I20260929 10:28:00.750682 20936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d6f3e811713410ca11d0720982b8a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d6f3e811713410ca11d0720982b8a20" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 40327 } }
I20260929 10:28:00.750823 20936 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d6f3e811713410ca11d0720982b8a20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:28:00.750922 20936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d6f3e811713410ca11d0720982b8a20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:28:00.751057 20936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d6f3e811713410ca11d0720982b8a20 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:00.752103 20936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d6f3e811713410ca11d0720982b8a20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d6f3e811713410ca11d0720982b8a20" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 40327 } }
I20260929 10:28:00.752503 20936 leader_election.cc:302] T 00000000000000000000000000000000 P 2d6f3e811713410ca11d0720982b8a20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d6f3e811713410ca11d0720982b8a20; no voters:
I20260929 10:28:00.752892 20936 leader_election.cc:288] T 00000000000000000000000000000000 P 2d6f3e811713410ca11d0720982b8a20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:28:00.753043 20941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d6f3e811713410ca11d0720982b8a20 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:00.753300 20941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d6f3e811713410ca11d0720982b8a20 [term 1 LEADER]: Becoming Leader. State: Replica: 2d6f3e811713410ca11d0720982b8a20, State: Running, Role: LEADER
I20260929 10:28:00.753682 20941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d6f3e811713410ca11d0720982b8a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d6f3e811713410ca11d0720982b8a20" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 40327 } }
I20260929 10:28:00.754391 20936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d6f3e811713410ca11d0720982b8a20 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:28:00.755501 20942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d6f3e811713410ca11d0720982b8a20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d6f3e811713410ca11d0720982b8a20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d6f3e811713410ca11d0720982b8a20" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 40327 } } }
I20260929 10:28:00.755630 20942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d6f3e811713410ca11d0720982b8a20 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:00.756029 20950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 29 10:28:00 dist-test-slave-9cj5 krb5kdc[20855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:00.758397 20943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d6f3e811713410ca11d0720982b8a20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d6f3e811713410ca11d0720982b8a20. Latest consensus state: current_term: 1 leader_uuid: "2d6f3e811713410ca11d0720982b8a20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d6f3e811713410ca11d0720982b8a20" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 40327 } } }
I20260929 10:28:00.758569 20943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d6f3e811713410ca11d0720982b8a20 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:00.759554 20950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:28:00.764407 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:00.739690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58612 (local address 127.20.92.62:40327)
0929 10:28:00.740202 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:00.740217 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:00.741400 (+ 1183us) server_negotiation.cc:407] Connection header received
0929 10:28:00.744925 (+ 3525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:00.744942 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:00.745311 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:00.745732 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:00.748369 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:00.750006 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:00.751629 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:00.752103 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:00.752480 (+ 377us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:00.757876 (+ 5396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:00.757883 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:00.757899 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:00.757935 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:00.760155 (+ 2220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:00.761305 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:00.761312 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:00.761317 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:00.761468 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:00.761961 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:00.761964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:00.761966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:00.762378 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:00.762611 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:00.763177 (+ 566us) server_negotiation.cc:299] Negotiation successful
0929 10:28:00.763459 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":153,"threads_started":1}
I20260929 10:28:00.766592 20950 catalog_manager.cc:1347] Generated new cluster ID: 20a7064bf39a463987299bccb2c8ff15
I20260929 10:28:00.766666 20950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:28:00.789404 20950 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:28:00.790797 20950 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:28:00.798255 20950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d6f3e811713410ca11d0720982b8a20: Generated new TSK 0
I20260929 10:28:00.799198 20950 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
I20260929 10:28:00.838795 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:0
--local_ip_for_outbound_sockets=127.20.92.1
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:40327
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:42707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:00.956559 20962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:00.956897 20962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:00.956976 20962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:00.961133 20962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:00.961251 20962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:00.961354 20962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:28:00.968560 20962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:42707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:40327
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:00.970000 20962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:00.971542 20962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:00.983151 20967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:00.983151 20968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:00.983256 20970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:00.983670 20962 server_base.cc:1034] running on GCE node
I20260929 10:28:00.984109 20962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:00.984782 20962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:00.986074 20962 hybrid_clock.cc:648] HybridClock initialized: now 1790677680986031 us; error 51 us; skew 500 ppm
Sep 29 10:28:00 dist-test-slave-9cj5 krb5kdc[20855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677680, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:00.989032 20962 init.cc:377] Logged in from keytab as kudu/127.20.92.1@KRBTEST.COM (short username kudu)
I20260929 10:28:00.990036 20962 webserver.cc:460] Webserver started at http://127.20.92.1:33469/ using document root <none> and password file <none>
I20260929 10:28:00.990572 20962 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:00.990631 20962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:00.990808 20962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:00.992983 20962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-0/data/instance:
uuid: "92c25c808e8b4973ae86af215104a924"
format_stamp: "Formatted at 2026-09-29 10:28:00 on dist-test-slave-9cj5"
server_key: "a5692c5c55e194549a251f5d7cc8a624"
server_key_iv: "782e58323b7b6ef81ea39f275f69d89d"
server_key_version: "encryptionkey@0"
I20260929 10:28:00.993520 20962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance:
uuid: "92c25c808e8b4973ae86af215104a924"
format_stamp: "Formatted at 2026-09-29 10:28:00 on dist-test-slave-9cj5"
server_key: "a5692c5c55e194549a251f5d7cc8a624"
server_key_iv: "782e58323b7b6ef81ea39f275f69d89d"
server_key_version: "encryptionkey@0"
I20260929 10:28:00.997628 20962 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 10:28:01.000447 20977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:01.002064 20962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:28:01.002182 20962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "92c25c808e8b4973ae86af215104a924"
format_stamp: "Formatted at 2026-09-29 10:28:00 on dist-test-slave-9cj5"
server_key: "a5692c5c55e194549a251f5d7cc8a624"
server_key_iv: "782e58323b7b6ef81ea39f275f69d89d"
server_key_version: "encryptionkey@0"
I20260929 10:28:01.002350 20962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:01.026646 20962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:01.030651 20962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:01.030885 20962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:01.031558 20962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:01.032495 20962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:01.032547 20962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:01.032629 20962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:01.032663 20962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:01.043052 20962 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:42061
I20260929 10:28:01.043075 21090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:42061 every 8 connection(s)
I20260929 10:28:01.044028 20962 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
I20260929 10:28:01.046396 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 20962
I20260929 10:28:01.046494 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance
I20260929 10:28:01.046787 20848 external_mini_cluster.cc:1428] Setting key 8f4306767fcbbe7eb00f357756e28c0e
Sep 29 10:28:01 dist-test-slave-9cj5 krb5kdc[20855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677680, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.92.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.2@KRBTEST.COM" created.
I20260929 10:28:01.060286 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:01.046083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:36465 (local address 127.20.92.62:40327)
0929 10:28:01.046256 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:01.046261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:01.047589 (+ 1328us) server_negotiation.cc:407] Connection header received
0929 10:28:01.048558 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:01.048562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:01.048617 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:01.048714 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:01.050063 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:01.050643 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:01.051911 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:01.052168 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:01.052292 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:01.055669 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:01.055693 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:01.055698 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:01.055729 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:01.057724 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:01.058382 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:01.058385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:01.058387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:01.058470 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:01.058844 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:01.058847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:01.058848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:01.059129 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:01.059226 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:01.059819 (+ 593us) server_negotiation.cc:299] Negotiation successful
0929 10:28:01.059985 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 10:28:01.061235 21093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:01.046577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40327 (local address 127.20.92.1:36465)
0929 10:28:01.047399 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:01.047473 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:01.048340 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:01.048880 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:01.048887 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:01.049262 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:01.049900 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:01.049918 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:01.050807 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:01.050811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:01.051766 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:01.051775 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:01.052333 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:01.052338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:01.052528 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:01.053209 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:01.053227 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:01.055329 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:01.057890 (+ 2561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:01.057896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:01.057910 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:01.058252 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:01.058554 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:01.058557 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:01.058558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:01.058707 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:01.059241 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:01.059247 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:01.059519 (+ 272us) client_negotiation.cc:769] Sending connection context
0929 10:28:01.059678 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 10:28:01.060094 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":153,"threads_started":1}
I20260929 10:28:01.063125 21091 heartbeater.cc:347] Connected to a master server at 127.20.92.62:40327
I20260929 10:28:01.063421 21091 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:01.063942 21091 heartbeater.cc:511] Master 127.20.92.62:40327 requested a full tablet report, sending...
I20260929 10:28:01.065543 20900 ts_manager.cc:194] Registered new tserver with Master: 92c25c808e8b4973ae86af215104a924 (127.20.92.1:42061)
I20260929 10:28:01.066855 20900 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.1@KRBTEST.COM'} at 127.20.92.1:36465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
I20260929 10:28:01.083415 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:0
--local_ip_for_outbound_sockets=127.20.92.2
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:40327
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:42707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:01.200096 21096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:01.200565 21096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:01.200666 21096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:01.205216 21096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:01.205331 21096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:01.205454 21096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:28:01.211712 21096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:42707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:40327
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:01.212931 21096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:01.214161 21096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:01.225934 21102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:01.225957 21101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:01.225992 21096 server_base.cc:1034] running on GCE node
W20260929 10:28:01.225934 21104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:01.226677 21096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:01.227422 21096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:01.228605 21096 hybrid_clock.cc:648] HybridClock initialized: now 1790677681228581 us; error 35 us; skew 500 ppm
Sep 29 10:28:01 dist-test-slave-9cj5 krb5kdc[20855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677681, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:01.232966 21096 init.cc:377] Logged in from keytab as kudu/127.20.92.2@KRBTEST.COM (short username kudu)
I20260929 10:28:01.234139 21096 webserver.cc:460] Webserver started at http://127.20.92.2:42935/ using document root <none> and password file <none>
I20260929 10:28:01.234767 21096 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:01.235311 21096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:01.235692 21096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:01.238993 21096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-1/data/instance:
uuid: "99b8497c5cde44a398014c708da3312b"
format_stamp: "Formatted at 2026-09-29 10:28:01 on dist-test-slave-9cj5"
server_key: "258d4965d0e5693a821300b497238e90"
server_key_iv: "be4812088cf767be3fbbf3c70b5a5c67"
server_key_version: "encryptionkey@0"
I20260929 10:28:01.239750 21096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance:
uuid: "99b8497c5cde44a398014c708da3312b"
format_stamp: "Formatted at 2026-09-29 10:28:01 on dist-test-slave-9cj5"
server_key: "258d4965d0e5693a821300b497238e90"
server_key_iv: "be4812088cf767be3fbbf3c70b5a5c67"
server_key_version: "encryptionkey@0"
I20260929 10:28:01.244292 21096 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:28:01.247454 21111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:01.249152 21096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 10:28:01.249366 21096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "99b8497c5cde44a398014c708da3312b"
format_stamp: "Formatted at 2026-09-29 10:28:01 on dist-test-slave-9cj5"
server_key: "258d4965d0e5693a821300b497238e90"
server_key_iv: "be4812088cf767be3fbbf3c70b5a5c67"
server_key_version: "encryptionkey@0"
I20260929 10:28:01.249547 21096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:01.276003 21096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:01.279927 21096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:01.280164 21096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:01.280835 21096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:01.281729 21096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:01.281844 21096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:01.281944 21096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:01.281993 21096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:01.294812 21096 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:33549
I20260929 10:28:01.294886 21224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:33549 every 8 connection(s)
I20260929 10:28:01.295861 21096 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
I20260929 10:28:01.302469 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 21096
I20260929 10:28:01.302587 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance
I20260929 10:28:01.302842 20848 external_mini_cluster.cc:1428] Setting key 0fa7634ffacf4310a8392a9ebd09a4ba
Sep 29 10:28:01 dist-test-slave-9cj5 krb5kdc[20855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677681, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:01.313078 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:01.298207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:55189 (local address 127.20.92.62:40327)
0929 10:28:01.298395 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:01.298403 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:01.299663 (+ 1260us) server_negotiation.cc:407] Connection header received
0929 10:28:01.300736 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:01.300740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:01.300801 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:01.300908 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:01.302451 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:01.303166 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:01.304639 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:01.304872 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:01.304991 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:01.308226 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:01.308234 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:01.308241 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:01.308283 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:01.310378 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:01.311079 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:01.311085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:01.311090 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:01.311209 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:01.311684 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:01.311692 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:01.311694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:01.311985 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:01.312106 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:01.312720 (+ 614us) server_negotiation.cc:299] Negotiation successful
0929 10:28:01.312871 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
WARNING: no policy specified for kudu/127.20.92.3@KRBTEST.COM; defaulting to no policy
I20260929 10:28:01.314131 21227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:01.298630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40327 (local address 127.20.92.2:55189)
0929 10:28:01.299482 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:01.299530 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:01.300491 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:01.301112 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:01.301120 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:01.301527 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:01.302135 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:01.302169 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:01.303336 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:01.303339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:01.304507 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:01.304518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:01.305043 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:01.305047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:01.305231 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:01.305961 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:01.305980 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:01.307971 (+ 1991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:01.310572 (+ 2601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:01.310580 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:01.310596 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:01.310953 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:01.311384 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:01.311387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:01.311389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:01.311563 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:01.312108 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:01.312114 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:01.312435 (+ 321us) client_negotiation.cc:769] Sending connection context
0929 10:28:01.312626 (+ 191us) client_negotiation.cc:240] Negotiation successful
0929 10:28:01.312956 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":134,"threads_started":1}
I20260929 10:28:01.315599 21225 heartbeater.cc:347] Connected to a master server at 127.20.92.62:40327
I20260929 10:28:01.315934 21225 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:01.316618 21225 heartbeater.cc:511] Master 127.20.92.62:40327 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.3@KRBTEST.COM" created.
I20260929 10:28:01.318307 20900 ts_manager.cc:194] Registered new tserver with Master: 99b8497c5cde44a398014c708da3312b (127.20.92.2:33549)
I20260929 10:28:01.318981 20900 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.2@KRBTEST.COM'} at 127.20.92.2:55189
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
I20260929 10:28:01.337445 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:0
--local_ip_for_outbound_sockets=127.20.92.3
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:40327
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:42707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:01.463140 21230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:01.463442 21230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:01.463554 21230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:01.468120 21230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:01.468202 21230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:01.468271 21230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:28:01.474180 21230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:42707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:40327
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:01.475317 21230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:01.476377 21230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:01.486832 21236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:01.486804 21238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:01.486804 21235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:01.487115 21230 server_base.cc:1034] running on GCE node
I20260929 10:28:01.487730 21230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:01.488562 21230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:01.489863 21230 hybrid_clock.cc:648] HybridClock initialized: now 1790677681489818 us; error 50 us; skew 500 ppm
Sep 29 10:28:01 dist-test-slave-9cj5 krb5kdc[20855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677681, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:01.493012 21230 init.cc:377] Logged in from keytab as kudu/127.20.92.3@KRBTEST.COM (short username kudu)
I20260929 10:28:01.494267 21230 webserver.cc:460] Webserver started at http://127.20.92.3:45069/ using document root <none> and password file <none>
I20260929 10:28:01.494866 21230 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:01.494952 21230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:01.495132 21230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:01.497193 21230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-2/data/instance:
uuid: "a15191ad75c1472080a605ed45f86679"
format_stamp: "Formatted at 2026-09-29 10:28:01 on dist-test-slave-9cj5"
server_key: "2ccec390907763ea65cd27461ceceafc"
server_key_iv: "224faaf989d22936c1d59ec9efd658f2"
server_key_version: "encryptionkey@0"
I20260929 10:28:01.497730 21230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance:
uuid: "a15191ad75c1472080a605ed45f86679"
format_stamp: "Formatted at 2026-09-29 10:28:01 on dist-test-slave-9cj5"
server_key: "2ccec390907763ea65cd27461ceceafc"
server_key_iv: "224faaf989d22936c1d59ec9efd658f2"
server_key_version: "encryptionkey@0"
I20260929 10:28:01.501957 21230 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 10:28:01.505066 21245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:01.506669 21230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:28:01.506809 21230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "a15191ad75c1472080a605ed45f86679"
format_stamp: "Formatted at 2026-09-29 10:28:01 on dist-test-slave-9cj5"
server_key: "2ccec390907763ea65cd27461ceceafc"
server_key_iv: "224faaf989d22936c1d59ec9efd658f2"
server_key_version: "encryptionkey@0"
I20260929 10:28:01.506942 21230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:01.535975 21230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:01.539587 21230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:01.539837 21230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:01.540511 21230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:01.541404 21230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:01.541452 21230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:01.541525 21230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:01.541563 21230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:01.551522 21230 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:37387
I20260929 10:28:01.551553 21358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:37387 every 8 connection(s)
I20260929 10:28:01.552496 21230 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:28:01.555464 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 21230
I20260929 10:28:01.555636 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance
I20260929 10:28:01.555922 20848 external_mini_cluster.cc:1428] Setting key 06e4e9baba5d49c04fe70d6c36c6c0d6
Sep 29 10:28:01 dist-test-slave-9cj5 krb5kdc[20855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677681, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:01.567788 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:01.554707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:54851 (local address 127.20.92.62:40327)
0929 10:28:01.554913 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:01.554920 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:01.556262 (+ 1342us) server_negotiation.cc:407] Connection header received
0929 10:28:01.557366 (+ 1104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:01.557369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:01.557426 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:01.557524 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:01.558941 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:01.559502 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:01.560862 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:01.561077 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:01.561170 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:01.563731 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:01.563734 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:01.563736 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:01.563761 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:01.565330 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:01.566069 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:01.566072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:01.566074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:01.566211 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:01.566593 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:01.566596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:01.566597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:01.566830 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:01.566926 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:01.567494 (+ 568us) server_negotiation.cc:299] Negotiation successful
0929 10:28:01.567612 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 10:28:01.568656 21361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:01.555254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40327 (local address 127.20.92.3:54851)
0929 10:28:01.556110 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:01.556151 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:01.557119 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:01.557712 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:01.557723 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:01.558160 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:01.558748 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:01.558772 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:01.559638 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:01.559641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:01.560724 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:01.560734 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:01.561251 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:01.561254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:01.561421 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:01.562107 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:01.562123 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:01.563556 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:01.565461 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:01.565467 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:01.565480 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:01.565945 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:01.566317 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:01.566320 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:01.566322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:01.566474 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:01.566945 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:01.566950 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:01.567211 (+ 261us) client_negotiation.cc:769] Sending connection context
0929 10:28:01.567386 (+ 175us) client_negotiation.cc:240] Negotiation successful
0929 10:28:01.567681 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":164,"threads_started":1}
I20260929 10:28:01.570181 21359 heartbeater.cc:347] Connected to a master server at 127.20.92.62:40327
I20260929 10:28:01.570497 21359 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:01.571125 21359 heartbeater.cc:511] Master 127.20.92.62:40327 requested a full tablet report, sending...
I20260929 10:28:01.572261 20900 ts_manager.cc:194] Registered new tserver with Master: a15191ad75c1472080a605ed45f86679 (127.20.92.3:37387)
I20260929 10:28:01.572970 20900 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.3@KRBTEST.COM'} at 127.20.92.3:54851
I20260929 10:28:01.583415 20848 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 10:28:01 dist-test-slave-9cj5 krb5kdc[20855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677681, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 10:28:01 dist-test-slave-9cj5 krb5kdc[20855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677681, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.92.1@KRBTEST.COM
I20260929 10:28:01.612406 21366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:01.601253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36598 (local address 127.20.92.1:42061)
0929 10:28:01.601561 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:01.601567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:01.601592 (+ 25us) server_negotiation.cc:407] Connection header received
0929 10:28:01.601662 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:01.601667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:01.601994 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:01.602118 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:01.602984 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:01.603765 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:01.604714 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:01.605139 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:01.605239 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:01.608391 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:01.608401 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:01.608414 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:01.608442 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:01.610348 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:01.610950 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:01.610957 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:01.610959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:01.611048 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:01.611310 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:01.611312 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:01.611313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:01.611568 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:01.611731 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:01.612035 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 10:28:01.612193 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":104,"threads_started":1}
W20260929 10:28:01.613492 21005 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36598
Sep 29 10:28:01 dist-test-slave-9cj5 krb5kdc[20855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 10:28:01 dist-test-slave-9cj5 krb5kdc[20855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.1@KRBTEST.COM
I20260929 10:28:01.641460 21366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:01.631987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36604 (local address 127.20.92.1:42061)
0929 10:28:01.632132 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:01.632137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:01.632205 (+ 68us) server_negotiation.cc:407] Connection header received
0929 10:28:01.632335 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:01.632337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:01.632386 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:01.632466 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:01.633304 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:01.633974 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:01.634954 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:01.635175 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:01.635271 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:01.638182 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:01.638186 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:01.638188 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:01.638215 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:01.639793 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:01.640252 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:01.640255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:01.640257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:01.640337 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:01.640620 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:01.640623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:01.640624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:01.640788 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:01.640901 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:01.641135 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 10:28:01.641237 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 10:28:01.642776 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 20962
I20260929 10:28:01.648577 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 21096
I20260929 10:28:01.654408 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 21230
I20260929 10:28:01.660063 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 20868
2026-09-29T10:28:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2366 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:28:01 dist-test-slave-9cj5 krb5kdc[21373](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:28:01 dist-test-slave-9cj5 krb5kdc[21373](info): set up 2 sockets
Sep 29 10:28:01 dist-test-slave-9cj5 krb5kdc[21373](info): commencing operation
krb5kdc: starting...
W20260929 10:28:02.758735 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:28:02 dist-test-slave-9cj5 krb5kdc[21373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:28:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:28:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
I20260929 10:28:02.904069 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:45475
--webserver_interface=127.20.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:45607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:45475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:03.031060 21386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:03.031387 21386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:03.031466 21386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:03.035763 21386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:28:03.035876 21386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:03.035915 21386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:03.035952 21386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:28:03.035995 21386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:28:03.041772 21386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:45607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:45475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:45475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:03.042999 21386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:03.044091 21386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:03.054082 21394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:03.054167 21386 server_base.cc:1034] running on GCE node
W20260929 10:28:03.054116 21391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:03.054101 21392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:03.055063 21386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:03.056360 21386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:03.057555 21386 hybrid_clock.cc:648] HybridClock initialized: now 1790677683057513 us; error 60 us; skew 500 ppm
Sep 29 10:28:03 dist-test-slave-9cj5 krb5kdc[21373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677683, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:03.065660 21386 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:28:03.066824 21386 webserver.cc:460] Webserver started at http://127.20.92.62:35913/ using document root <none> and password file <none>
I20260929 10:28:03.067478 21386 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:03.067554 21386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:03.067751 21386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:03.070190 21386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "67a0b14942984dbba9e85241cd8fd653"
format_stamp: "Formatted at 2026-09-29 10:28:03 on dist-test-slave-9cj5"
server_key: "30c3c5dbbbf71e4b330adba21d42b399"
server_key_iv: "c5634adaed87425208369dbcfd7a7b98"
server_key_version: "encryptionkey@0"
I20260929 10:28:03.071116 21386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "67a0b14942984dbba9e85241cd8fd653"
format_stamp: "Formatted at 2026-09-29 10:28:03 on dist-test-slave-9cj5"
server_key: "30c3c5dbbbf71e4b330adba21d42b399"
server_key_iv: "c5634adaed87425208369dbcfd7a7b98"
server_key_version: "encryptionkey@0"
I20260929 10:28:03.076915 21386 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 10:28:03.079984 21401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:03.081944 21386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:28:03.082099 21386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "67a0b14942984dbba9e85241cd8fd653"
format_stamp: "Formatted at 2026-09-29 10:28:03 on dist-test-slave-9cj5"
server_key: "30c3c5dbbbf71e4b330adba21d42b399"
server_key_iv: "c5634adaed87425208369dbcfd7a7b98"
server_key_version: "encryptionkey@0"
I20260929 10:28:03.082286 21386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:03.115145 21386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:03.118707 21386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:03.118872 21386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:03.126533 21386 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:45475
I20260929 10:28:03.126557 21453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:45475 every 8 connection(s)
I20260929 10:28:03.127554 21386 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:28:03.130553 21454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:03.133008 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 21386
I20260929 10:28:03.133126 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:28:03.133388 20848 external_mini_cluster.cc:1428] Setting key 1ae9eff191dd34611920f188376899b3
I20260929 10:28:03.136960 21454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67a0b14942984dbba9e85241cd8fd653: Bootstrap starting.
I20260929 10:28:03.139829 21454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67a0b14942984dbba9e85241cd8fd653: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:03.140630 21454 log.cc:824] T 00000000000000000000000000000000 P 67a0b14942984dbba9e85241cd8fd653: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:03.142699 21454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67a0b14942984dbba9e85241cd8fd653: No bootstrap required, opened a new log
Sep 29 10:28:03 dist-test-slave-9cj5 krb5kdc[21373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:03.146960 21454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67a0b14942984dbba9e85241cd8fd653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a0b14942984dbba9e85241cd8fd653" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45475 } }
I20260929 10:28:03.147509 21454 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67a0b14942984dbba9e85241cd8fd653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:03.147709 21454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67a0b14942984dbba9e85241cd8fd653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a0b14942984dbba9e85241cd8fd653, State: Initialized, Role: FOLLOWER
I20260929 10:28:03.148409 21454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67a0b14942984dbba9e85241cd8fd653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a0b14942984dbba9e85241cd8fd653" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45475 } }
I20260929 10:28:03.148835 21454 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67a0b14942984dbba9e85241cd8fd653 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:28:03.149004 21454 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67a0b14942984dbba9e85241cd8fd653 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:28:03.149179 21454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67a0b14942984dbba9e85241cd8fd653 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:03.150497 21454 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67a0b14942984dbba9e85241cd8fd653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a0b14942984dbba9e85241cd8fd653" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45475 } }
I20260929 10:28:03.150801 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:03.135007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56926 (local address 127.20.92.62:45475)
0929 10:28:03.135507 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:03.135523 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:03.135561 (+ 38us) server_negotiation.cc:407] Connection header received
0929 10:28:03.136219 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:03.136237 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:03.136592 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:03.136962 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:03.138016 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:03.139281 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:03.140275 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:03.140818 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:03.140982 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:03.143742 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:03.143748 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:03.143763 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:03.143793 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:03.145672 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:03.146938 (+ 1266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:03.146945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:03.146954 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:03.147157 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:03.147535 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:03.147544 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:03.147545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:03.148409 (+ 864us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:03.148961 (+ 552us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:03.149327 (+ 366us) server_negotiation.cc:299] Negotiation successful
0929 10:28:03.149661 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":141,"threads_started":1}
I20260929 10:28:03.151013 21454 leader_election.cc:302] T 00000000000000000000000000000000 P 67a0b14942984dbba9e85241cd8fd653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67a0b14942984dbba9e85241cd8fd653; no voters:
I20260929 10:28:03.151542 21454 leader_election.cc:288] T 00000000000000000000000000000000 P 67a0b14942984dbba9e85241cd8fd653 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:28:03.152074 21459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67a0b14942984dbba9e85241cd8fd653 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:03.152449 21459 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67a0b14942984dbba9e85241cd8fd653 [term 1 LEADER]: Becoming Leader. State: Replica: 67a0b14942984dbba9e85241cd8fd653, State: Running, Role: LEADER
I20260929 10:28:03.153119 21454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67a0b14942984dbba9e85241cd8fd653 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:28:03.153084 21459 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67a0b14942984dbba9e85241cd8fd653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a0b14942984dbba9e85241cd8fd653" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45475 } }
I20260929 10:28:03.155150 21461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67a0b14942984dbba9e85241cd8fd653 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67a0b14942984dbba9e85241cd8fd653. Latest consensus state: current_term: 1 leader_uuid: "67a0b14942984dbba9e85241cd8fd653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a0b14942984dbba9e85241cd8fd653" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45475 } } }
I20260929 10:28:03.155318 21461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67a0b14942984dbba9e85241cd8fd653 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:03.155797 21460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67a0b14942984dbba9e85241cd8fd653 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67a0b14942984dbba9e85241cd8fd653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a0b14942984dbba9e85241cd8fd653" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45475 } } }
I20260929 10:28:03.155967 21460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67a0b14942984dbba9e85241cd8fd653 [sys.catalog]: This master's current role is: LEADER
W20260929 10:28:03.159531 21473 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 67a0b14942984dbba9e85241cd8fd653: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:28:03.159631 21473 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:28:03.159741 21474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:28:03.160547 21474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:28:03.166793 21474 catalog_manager.cc:1347] Generated new cluster ID: 7782ac5770eb4acb8eabbe71fced232b
I20260929 10:28:03.166877 21474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:28:03.178376 21474 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:28:03.179360 21474 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:28:03.192664 21474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67a0b14942984dbba9e85241cd8fd653: Generated new TSK 0
I20260929 10:28:03.193495 21474 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
I20260929 10:28:03.229944 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:0
--local_ip_for_outbound_sockets=127.20.92.1
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:45475
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:45607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:03.356091 21480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:03.356397 21480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:03.356447 21480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:03.360749 21480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:03.360831 21480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:03.360903 21480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:28:03.366595 21480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:45607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:45475
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:03.367813 21480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:03.368897 21480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:03.381395 21485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:03.381249 21486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:03.381397 21488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:03.381899 21480 server_base.cc:1034] running on GCE node
I20260929 10:28:03.382313 21480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:03.382995 21480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:03.384215 21480 hybrid_clock.cc:648] HybridClock initialized: now 1790677683384201 us; error 29 us; skew 500 ppm
Sep 29 10:28:03 dist-test-slave-9cj5 krb5kdc[21373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677683, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:03.387704 21480 init.cc:377] Logged in from keytab as kudu/127.20.92.1@KRBTEST.COM (short username kudu)
I20260929 10:28:03.388859 21480 webserver.cc:460] Webserver started at http://127.20.92.1:32945/ using document root <none> and password file <none>
I20260929 10:28:03.389473 21480 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:03.389575 21480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:03.389873 21480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:03.392050 21480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-0/data/instance:
uuid: "fb9db56f0cc34c7cb653306f3dac1f05"
format_stamp: "Formatted at 2026-09-29 10:28:03 on dist-test-slave-9cj5"
server_key: "63eff24ba8d725a13330997dfb59a646"
server_key_iv: "67417adb1a9d2dd768124bfdc3970f26"
server_key_version: "encryptionkey@0"
I20260929 10:28:03.392660 21480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance:
uuid: "fb9db56f0cc34c7cb653306f3dac1f05"
format_stamp: "Formatted at 2026-09-29 10:28:03 on dist-test-slave-9cj5"
server_key: "63eff24ba8d725a13330997dfb59a646"
server_key_iv: "67417adb1a9d2dd768124bfdc3970f26"
server_key_version: "encryptionkey@0"
I20260929 10:28:03.396847 21480 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 10:28:03.399821 21495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:03.401319 21480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:28:03.401512 21480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "fb9db56f0cc34c7cb653306f3dac1f05"
format_stamp: "Formatted at 2026-09-29 10:28:03 on dist-test-slave-9cj5"
server_key: "63eff24ba8d725a13330997dfb59a646"
server_key_iv: "67417adb1a9d2dd768124bfdc3970f26"
server_key_version: "encryptionkey@0"
I20260929 10:28:03.401660 21480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:03.430840 21480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:03.435129 21480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:03.435392 21480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:03.436110 21480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:03.437052 21480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:03.437132 21480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:03.437214 21480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:03.437273 21480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:03.448588 21480 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:39489
I20260929 10:28:03.448642 21608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:39489 every 8 connection(s)
I20260929 10:28:03.449638 21480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
I20260929 10:28:03.458489 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 21480
I20260929 10:28:03.458635 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance
I20260929 10:28:03.459029 20848 external_mini_cluster.cc:1428] Setting key 49c5d86182fd0f8b191ab357d1738c6c
Sep 29 10:28:03 dist-test-slave-9cj5 krb5kdc[21373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677683, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:03.467324 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:03.451813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:33361 (local address 127.20.92.62:45475)
0929 10:28:03.452041 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:03.452045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:03.453036 (+ 991us) server_negotiation.cc:407] Connection header received
0929 10:28:03.454430 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:03.454434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:03.454505 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:03.454619 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:03.456039 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:03.456640 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:03.458111 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:03.458574 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:03.458697 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:03.462501 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:03.462509 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:03.462516 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:03.462558 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:03.464467 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:03.465263 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:03.465267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:03.465268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:03.465361 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:03.465959 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:03.465962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:03.465964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:03.466178 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:03.466279 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:03.467072 (+ 793us) server_negotiation.cc:299] Negotiation successful
0929 10:28:03.467194 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260929 10:28:03.468693 21611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:03.452251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45475 (local address 127.20.92.1:33361)
0929 10:28:03.452790 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:03.452896 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:03.454150 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:03.454859 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:03.454873 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:03.455294 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:03.455864 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:03.455880 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:03.456789 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:03.456793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:03.457941 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:03.457959 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:03.458908 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:03.458915 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:03.459153 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:03.459997 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:03.460022 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:03.462277 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:03.464615 (+ 2338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:03.464627 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:03.464651 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:03.465102 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:03.465517 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:03.465524 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:03.465529 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:03.465796 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:03.466318 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:03.466328 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:03.466724 (+ 396us) client_negotiation.cc:769] Sending connection context
0929 10:28:03.466987 (+ 263us) client_negotiation.cc:240] Negotiation successful
0929 10:28:03.467500 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":121,"threads_started":1}
WARNING: no policy specified for kudu/127.20.92.2@KRBTEST.COM; defaulting to no policy
I20260929 10:28:03.470360 21609 heartbeater.cc:347] Connected to a master server at 127.20.92.62:45475
I20260929 10:28:03.470606 21609 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:03.471115 21609 heartbeater.cc:511] Master 127.20.92.62:45475 requested a full tablet report, sending...
I20260929 10:28:03.472756 21418 ts_manager.cc:194] Registered new tserver with Master: fb9db56f0cc34c7cb653306f3dac1f05 (127.20.92.1:39489)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.2@KRBTEST.COM" created.
I20260929 10:28:03.474169 21418 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.1@KRBTEST.COM'} at 127.20.92.1:33361
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
I20260929 10:28:03.495967 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:0
--local_ip_for_outbound_sockets=127.20.92.2
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:45475
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:45607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:03.618741 21614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:03.619057 21614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:03.619107 21614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:03.623524 21614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:03.623605 21614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:03.623672 21614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:28:03.629211 21614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:45607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:45475
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:03.630424 21614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:03.631448 21614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:28:03.641300 21614 server_base.cc:1034] running on GCE node
W20260929 10:28:03.641441 21620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:03.641541 21622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:03.641362 21619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:03.642031 21614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:03.642736 21614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:03.643999 21614 hybrid_clock.cc:648] HybridClock initialized: now 1790677683643966 us; error 54 us; skew 500 ppm
Sep 29 10:28:03 dist-test-slave-9cj5 krb5kdc[21373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677683, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:03.647027 21614 init.cc:377] Logged in from keytab as kudu/127.20.92.2@KRBTEST.COM (short username kudu)
I20260929 10:28:03.648047 21614 webserver.cc:460] Webserver started at http://127.20.92.2:40965/ using document root <none> and password file <none>
I20260929 10:28:03.648630 21614 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:03.648725 21614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:03.648965 21614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:03.651170 21614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-1/data/instance:
uuid: "4d71ab203d844ab2b31248796b67d76e"
format_stamp: "Formatted at 2026-09-29 10:28:03 on dist-test-slave-9cj5"
server_key: "982dbacbac4d466d3b0062b1f1e5bcde"
server_key_iv: "9a9646de5cb4a0eb1989bde226cecb88"
server_key_version: "encryptionkey@0"
I20260929 10:28:03.651736 21614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance:
uuid: "4d71ab203d844ab2b31248796b67d76e"
format_stamp: "Formatted at 2026-09-29 10:28:03 on dist-test-slave-9cj5"
server_key: "982dbacbac4d466d3b0062b1f1e5bcde"
server_key_iv: "9a9646de5cb4a0eb1989bde226cecb88"
server_key_version: "encryptionkey@0"
I20260929 10:28:03.655965 21614 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:28:03.658510 21629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:03.659684 21614 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:28:03.659826 21614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "4d71ab203d844ab2b31248796b67d76e"
format_stamp: "Formatted at 2026-09-29 10:28:03 on dist-test-slave-9cj5"
server_key: "982dbacbac4d466d3b0062b1f1e5bcde"
server_key_iv: "9a9646de5cb4a0eb1989bde226cecb88"
server_key_version: "encryptionkey@0"
I20260929 10:28:03.659941 21614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:03.684026 21614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:03.687913 21614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:03.688143 21614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:03.688803 21614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:03.689695 21614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:03.689805 21614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:03.689903 21614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:03.689957 21614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:03.700081 21614 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:33733
I20260929 10:28:03.700100 21742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:33733 every 8 connection(s)
I20260929 10:28:03.701210 21614 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
I20260929 10:28:03.703815 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 21614
I20260929 10:28:03.703922 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance
I20260929 10:28:03.704253 20848 external_mini_cluster.cc:1428] Setting key b20790e186676c47112a489bdbcf96f4
Sep 29 10:28:03 dist-test-slave-9cj5 krb5kdc[21373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677683, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.92.3@KRBTEST.COM; defaulting to no policy
I20260929 10:28:03.717370 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:03.703396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:50431 (local address 127.20.92.62:45475)
0929 10:28:03.703561 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:03.703565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:03.704646 (+ 1081us) server_negotiation.cc:407] Connection header received
0929 10:28:03.706615 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:03.706618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:03.706672 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:03.706879 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:03.708193 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:03.708725 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:03.709988 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:03.710216 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:03.710352 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:03.713435 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:03.713439 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:03.713441 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:03.713465 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:03.715108 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:03.715678 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:03.715681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:03.715684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:03.715771 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:03.716144 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:03.716149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:03.716151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:03.716363 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:03.716472 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:03.717048 (+ 576us) server_negotiation.cc:299] Negotiation successful
0929 10:28:03.717181 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.3@KRBTEST.COM" created.
I20260929 10:28:03.718279 21745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:03.703858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45475 (local address 127.20.92.2:50431)
0929 10:28:03.704495 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:03.704540 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:03.706364 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:03.707093 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:03.707100 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:03.707517 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:03.708033 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:03.708046 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:03.708860 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:03.708864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:03.709863 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:03.709874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:03.710409 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:03.710413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:03.710584 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:03.711332 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:03.711350 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:03.713236 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:03.715242 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:03.715249 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:03.715267 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:03.715563 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:03.715866 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:03.715870 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:03.715872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:03.716041 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:03.716474 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:03.716483 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:03.716784 (+ 301us) client_negotiation.cc:769] Sending connection context
0929 10:28:03.716991 (+ 207us) client_negotiation.cc:240] Negotiation successful
0929 10:28:03.717332 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":165,"threads_started":1}
I20260929 10:28:03.719705 21743 heartbeater.cc:347] Connected to a master server at 127.20.92.62:45475
I20260929 10:28:03.720007 21743 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:03.720517 21743 heartbeater.cc:511] Master 127.20.92.62:45475 requested a full tablet report, sending...
I20260929 10:28:03.721606 21418 ts_manager.cc:194] Registered new tserver with Master: 4d71ab203d844ab2b31248796b67d76e (127.20.92.2:33733)
I20260929 10:28:03.722506 21418 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.2@KRBTEST.COM'} at 127.20.92.2:50431
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
I20260929 10:28:03.737890 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:0
--local_ip_for_outbound_sockets=127.20.92.3
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:45475
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:45607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:03.856524 21748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:03.856791 21748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:03.856835 21748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:03.860954 21748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:03.861027 21748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:03.861090 21748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:28:03.866667 21748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:45607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:45475
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:03.867776 21748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:03.868806 21748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:03.878739 21756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:03.878724 21754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:03.878954 21753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:03.879122 21748 server_base.cc:1034] running on GCE node
I20260929 10:28:03.879647 21748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:03.880327 21748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:03.881547 21748 hybrid_clock.cc:648] HybridClock initialized: now 1790677683881506 us; error 54 us; skew 500 ppm
Sep 29 10:28:03 dist-test-slave-9cj5 krb5kdc[21373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677683, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:03.884598 21748 init.cc:377] Logged in from keytab as kudu/127.20.92.3@KRBTEST.COM (short username kudu)
I20260929 10:28:03.885540 21748 webserver.cc:460] Webserver started at http://127.20.92.3:42305/ using document root <none> and password file <none>
I20260929 10:28:03.886312 21748 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:03.886408 21748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:03.886686 21748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:03.889098 21748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-2/data/instance:
uuid: "17f48f89663e4bc8a1d48bf8e16edd41"
format_stamp: "Formatted at 2026-09-29 10:28:03 on dist-test-slave-9cj5"
server_key: "b1de654602ac9b127183ba03af7ff4c1"
server_key_iv: "dbc1e554cf7bda57f7dc38dd36a74364"
server_key_version: "encryptionkey@0"
I20260929 10:28:03.889676 21748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance:
uuid: "17f48f89663e4bc8a1d48bf8e16edd41"
format_stamp: "Formatted at 2026-09-29 10:28:03 on dist-test-slave-9cj5"
server_key: "b1de654602ac9b127183ba03af7ff4c1"
server_key_iv: "dbc1e554cf7bda57f7dc38dd36a74364"
server_key_version: "encryptionkey@0"
I20260929 10:28:03.893617 21748 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 10:28:03.896278 21763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:03.897711 21748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:28:03.897951 21748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "17f48f89663e4bc8a1d48bf8e16edd41"
format_stamp: "Formatted at 2026-09-29 10:28:03 on dist-test-slave-9cj5"
server_key: "b1de654602ac9b127183ba03af7ff4c1"
server_key_iv: "dbc1e554cf7bda57f7dc38dd36a74364"
server_key_version: "encryptionkey@0"
I20260929 10:28:03.898095 21748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:03.913614 21748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:03.917120 21748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:03.917351 21748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:03.918082 21748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:03.918947 21748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:03.919020 21748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:03.919093 21748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:03.919144 21748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:03.928853 21748 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:38319
I20260929 10:28:03.928905 21876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:38319 every 8 connection(s)
I20260929 10:28:03.929899 21748 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:28:03.934684 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 21748
I20260929 10:28:03.934787 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance
I20260929 10:28:03.935137 20848 external_mini_cluster.cc:1428] Setting key 9bf44f6c2886b1385ba990298555deeb
Sep 29 10:28:03 dist-test-slave-9cj5 krb5kdc[21373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677683, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:03.946823 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:03.932055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:42533 (local address 127.20.92.62:45475)
0929 10:28:03.932210 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:03.932214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:03.932991 (+ 777us) server_negotiation.cc:407] Connection header received
0929 10:28:03.934351 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:03.934355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:03.934418 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:03.934540 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:03.936049 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:03.936805 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:03.938419 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:03.938642 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:03.938737 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:03.941624 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:03.941627 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:03.941630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:03.941655 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:03.943543 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:03.944280 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:03.944283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:03.944284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:03.944401 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:03.944966 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:03.944970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:03.944971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:03.945557 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:03.945658 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:03.946512 (+ 854us) server_negotiation.cc:299] Negotiation successful
0929 10:28:03.946664 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 10:28:03.947777 21879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:03.932304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45475 (local address 127.20.92.3:42533)
0929 10:28:03.932770 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:03.932866 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:03.934103 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:03.934724 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:03.934731 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:03.935222 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:03.935881 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:03.935905 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:03.937117 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:03.937121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:03.938289 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:03.938300 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:03.938865 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:03.938868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:03.939039 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:03.939670 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:03.939689 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:03.941408 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:03.943768 (+ 2360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:03.943774 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:03.943789 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:03.944153 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:03.944526 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:03.944529 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:03.944531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:03.944803 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:03.945896 (+ 1093us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:03.945904 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:03.946191 (+ 287us) client_negotiation.cc:769] Sending connection context
0929 10:28:03.946411 (+ 220us) client_negotiation.cc:240] Negotiation successful
0929 10:28:03.946716 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":103,"threads_started":1}
I20260929 10:28:03.949182 21877 heartbeater.cc:347] Connected to a master server at 127.20.92.62:45475
I20260929 10:28:03.949484 21877 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:03.950279 21877 heartbeater.cc:511] Master 127.20.92.62:45475 requested a full tablet report, sending...
I20260929 10:28:03.951581 21417 ts_manager.cc:194] Registered new tserver with Master: 17f48f89663e4bc8a1d48bf8e16edd41 (127.20.92.3:38319)
I20260929 10:28:03.952380 21417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.3@KRBTEST.COM'} at 127.20.92.3:42533
I20260929 10:28:03.963500 20848 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:28:03.976792 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:03.967740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56928 (local address 127.20.92.62:45475)
0929 10:28:03.968043 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:03.968049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:03.968250 (+ 201us) server_negotiation.cc:407] Connection header received
0929 10:28:03.968438 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:03.968442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:03.968498 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:03.968609 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:03.969456 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:03.970100 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:03.971080 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:03.971332 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:03.971458 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:03.973032 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:03.973036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:03.973039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:03.973108 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:03.974787 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:03.975298 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:03.975302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:03.975304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:03.975394 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:03.975697 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:03.975704 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:03.975709 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:03.975964 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:03.976133 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:03.976474 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 10:28:03.976609 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198}
I20260929 10:28:03.983255 21417 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:28:03.986958 21417 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:28:04.004372 21891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:03.997773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57316 (local address 127.20.92.3:38319)
0929 10:28:03.998582 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:03.998597 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:03.998622 (+ 25us) server_negotiation.cc:407] Connection header received
0929 10:28:03.998917 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:03.998923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:03.999130 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:03.999295 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:03.999898 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:04.000722 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:04.002903 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:04.003573 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:04.003739 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:04.003932 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:04.004076 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 10:28:04.004182 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":163,"threads_started":1}
I20260929 10:28:04.004370 21889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:03.997365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:38319 (local address 127.0.0.1:57316)
0929 10:28:03.998200 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:03.998216 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:03.998317 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:03.999478 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:03.999482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:03.999515 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:03.999732 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:03.999739 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:04.000890 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:04.000903 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:04.002733 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:04.002745 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:04.003730 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:04.003734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:04.003849 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:04.003891 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 10:28:04.004028 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 10:28:04.004182 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"mutex_wait_us":193,"spinlock_wait_cycles":9088,"thread_start_us":122,"threads_started":1}
I20260929 10:28:04.005044 21893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:03.998326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46374 (local address 127.20.92.1:39489)
0929 10:28:03.998935 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:03.998942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:03.998964 (+ 22us) server_negotiation.cc:407] Connection header received
0929 10:28:03.999034 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:03.999039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:03.999200 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:03.999348 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:03.999881 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:04.000954 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:04.002809 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:04.003814 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:04.003945 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:04.004107 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:04.004665 (+ 558us) server_negotiation.cc:299] Negotiation successful
0929 10:28:04.004782 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":98,"threads_started":1}
I20260929 10:28:04.005904 21890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:03.997469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57570 (local address 127.20.92.2:33733)
0929 10:28:03.998583 (+ 1114us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:03.998597 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:03.998633 (+ 36us) server_negotiation.cc:407] Connection header received
0929 10:28:03.998716 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:03.998726 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:03.998966 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:03.999125 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:03.999593 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:04.000772 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:04.002995 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:04.004106 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:04.004224 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:04.004371 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:04.005494 (+ 1123us) server_negotiation.cc:299] Negotiation successful
0929 10:28:04.005725 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1013,"thread_start_us":120,"threads_started":1}
I20260929 10:28:04.006278 21888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:03.997313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:33733 (local address 127.0.0.1:57570)
0929 10:28:03.998053 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:03.998113 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:03.998277 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:03.999337 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:03.999343 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:03.999364 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:03.999470 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:03.999475 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:04.000923 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:04.000926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:04.002867 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:04.002877 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:04.004436 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:04.004440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:04.004807 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:04.004841 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 10:28:04.005045 (+ 204us) client_negotiation.cc:240] Negotiation successful
0929 10:28:04.006102 (+ 1057us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":430,"thread_start_us":109,"threads_started":1}
I20260929 10:28:04.006572 21892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:03.998160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:39489 (local address 127.0.0.1:46374)
0929 10:28:03.998688 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:03.998701 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:03.998779 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:03.999478 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:03.999482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:03.999517 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:03.999724 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:03.999732 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:04.001063 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:04.001070 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:04.002675 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:04.002685 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:04.004201 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:04.004206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:04.004313 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:04.004338 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 10:28:04.004419 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 10:28:04.006446 (+ 2027us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":1233,"spinlock_wait_cycles":4096,"thread_start_us":111,"threads_started":1}
I20260929 10:28:04.008383 21811 tablet_service.cc:1467] Processing CreateTablet for tablet 4bb305fb28384540863cdce02aed60dc (DEFAULT_TABLE table=test-table [id=94dbe6ebe6f14bb580d9f8115951d448]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:04.008989 21677 tablet_service.cc:1467] Processing CreateTablet for tablet 4bb305fb28384540863cdce02aed60dc (DEFAULT_TABLE table=test-table [id=94dbe6ebe6f14bb580d9f8115951d448]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:04.009609 21811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bb305fb28384540863cdce02aed60dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:04.009495 21543 tablet_service.cc:1467] Processing CreateTablet for tablet 4bb305fb28384540863cdce02aed60dc (DEFAULT_TABLE table=test-table [id=94dbe6ebe6f14bb580d9f8115951d448]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:04.010035 21677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bb305fb28384540863cdce02aed60dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:04.010447 21543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bb305fb28384540863cdce02aed60dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:04.016328 21894 tablet_bootstrap.cc:492] T 4bb305fb28384540863cdce02aed60dc P 4d71ab203d844ab2b31248796b67d76e: Bootstrap starting.
I20260929 10:28:04.018112 21896 tablet_bootstrap.cc:492] T 4bb305fb28384540863cdce02aed60dc P fb9db56f0cc34c7cb653306f3dac1f05: Bootstrap starting.
I20260929 10:28:04.019438 21894 tablet_bootstrap.cc:654] T 4bb305fb28384540863cdce02aed60dc P 4d71ab203d844ab2b31248796b67d76e: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:04.020349 21894 log.cc:824] T 4bb305fb28384540863cdce02aed60dc P 4d71ab203d844ab2b31248796b67d76e: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:04.020429 21896 tablet_bootstrap.cc:654] T 4bb305fb28384540863cdce02aed60dc P fb9db56f0cc34c7cb653306f3dac1f05: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:04.021184 21895 tablet_bootstrap.cc:492] T 4bb305fb28384540863cdce02aed60dc P 17f48f89663e4bc8a1d48bf8e16edd41: Bootstrap starting.
I20260929 10:28:04.021234 21896 log.cc:824] T 4bb305fb28384540863cdce02aed60dc P fb9db56f0cc34c7cb653306f3dac1f05: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:04.023448 21896 tablet_bootstrap.cc:492] T 4bb305fb28384540863cdce02aed60dc P fb9db56f0cc34c7cb653306f3dac1f05: No bootstrap required, opened a new log
I20260929 10:28:04.023465 21895 tablet_bootstrap.cc:654] T 4bb305fb28384540863cdce02aed60dc P 17f48f89663e4bc8a1d48bf8e16edd41: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:04.023670 21896 ts_tablet_manager.cc:1397] T 4bb305fb28384540863cdce02aed60dc P fb9db56f0cc34c7cb653306f3dac1f05: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 10:28:04.024669 21895 log.cc:824] T 4bb305fb28384540863cdce02aed60dc P 17f48f89663e4bc8a1d48bf8e16edd41: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:04.024808 21894 tablet_bootstrap.cc:492] T 4bb305fb28384540863cdce02aed60dc P 4d71ab203d844ab2b31248796b67d76e: No bootstrap required, opened a new log
I20260929 10:28:04.025029 21894 ts_tablet_manager.cc:1397] T 4bb305fb28384540863cdce02aed60dc P 4d71ab203d844ab2b31248796b67d76e: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260929 10:28:04.026893 21895 tablet_bootstrap.cc:492] T 4bb305fb28384540863cdce02aed60dc P 17f48f89663e4bc8a1d48bf8e16edd41: No bootstrap required, opened a new log
I20260929 10:28:04.027091 21895 ts_tablet_manager.cc:1397] T 4bb305fb28384540863cdce02aed60dc P 17f48f89663e4bc8a1d48bf8e16edd41: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 10:28:04.029474 21894 raft_consensus.cc:348] T 4bb305fb28384540863cdce02aed60dc P 4d71ab203d844ab2b31248796b67d76e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d71ab203d844ab2b31248796b67d76e" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 33733 } } peers { permanent_uuid: "17f48f89663e4bc8a1d48bf8e16edd41" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38319 } } peers { permanent_uuid: "fb9db56f0cc34c7cb653306f3dac1f05" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39489 } }
I20260929 10:28:04.029819 21894 raft_consensus.cc:374] T 4bb305fb28384540863cdce02aed60dc P 4d71ab203d844ab2b31248796b67d76e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:04.029875 21894 raft_consensus.cc:729] T 4bb305fb28384540863cdce02aed60dc P 4d71ab203d844ab2b31248796b67d76e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d71ab203d844ab2b31248796b67d76e, State: Initialized, Role: FOLLOWER
I20260929 10:28:04.029474 21896 raft_consensus.cc:348] T 4bb305fb28384540863cdce02aed60dc P fb9db56f0cc34c7cb653306f3dac1f05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d71ab203d844ab2b31248796b67d76e" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 33733 } } peers { permanent_uuid: "17f48f89663e4bc8a1d48bf8e16edd41" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38319 } } peers { permanent_uuid: "fb9db56f0cc34c7cb653306f3dac1f05" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39489 } }
I20260929 10:28:04.030035 21896 raft_consensus.cc:374] T 4bb305fb28384540863cdce02aed60dc P fb9db56f0cc34c7cb653306f3dac1f05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:04.030122 21896 raft_consensus.cc:729] T 4bb305fb28384540863cdce02aed60dc P fb9db56f0cc34c7cb653306f3dac1f05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb9db56f0cc34c7cb653306f3dac1f05, State: Initialized, Role: FOLLOWER
I20260929 10:28:04.030440 21894 consensus_queue.cc:260] T 4bb305fb28384540863cdce02aed60dc P 4d71ab203d844ab2b31248796b67d76e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d71ab203d844ab2b31248796b67d76e" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 33733 } } peers { permanent_uuid: "17f48f89663e4bc8a1d48bf8e16edd41" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38319 } } peers { permanent_uuid: "fb9db56f0cc34c7cb653306f3dac1f05" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39489 } }
I20260929 10:28:04.030571 21896 consensus_queue.cc:260] T 4bb305fb28384540863cdce02aed60dc P fb9db56f0cc34c7cb653306f3dac1f05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d71ab203d844ab2b31248796b67d76e" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 33733 } } peers { permanent_uuid: "17f48f89663e4bc8a1d48bf8e16edd41" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38319 } } peers { permanent_uuid: "fb9db56f0cc34c7cb653306f3dac1f05" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39489 } }
I20260929 10:28:04.031220 21743 heartbeater.cc:503] Master 127.20.92.62:45475 was elected leader, sending a full tablet report...
I20260929 10:28:04.031260 21895 raft_consensus.cc:348] T 4bb305fb28384540863cdce02aed60dc P 17f48f89663e4bc8a1d48bf8e16edd41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d71ab203d844ab2b31248796b67d76e" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 33733 } } peers { permanent_uuid: "17f48f89663e4bc8a1d48bf8e16edd41" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38319 } } peers { permanent_uuid: "fb9db56f0cc34c7cb653306f3dac1f05" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39489 } }
I20260929 10:28:04.031548 21895 raft_consensus.cc:374] T 4bb305fb28384540863cdce02aed60dc P 17f48f89663e4bc8a1d48bf8e16edd41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:04.031605 21895 raft_consensus.cc:729] T 4bb305fb28384540863cdce02aed60dc P 17f48f89663e4bc8a1d48bf8e16edd41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17f48f89663e4bc8a1d48bf8e16edd41, State: Initialized, Role: FOLLOWER
I20260929 10:28:04.031600 21894 ts_tablet_manager.cc:1428] T 4bb305fb28384540863cdce02aed60dc P 4d71ab203d844ab2b31248796b67d76e: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260929 10:28:04.031800 21896 ts_tablet_manager.cc:1428] T 4bb305fb28384540863cdce02aed60dc P fb9db56f0cc34c7cb653306f3dac1f05: Time spent starting tablet: real 0.008s user 0.002s sys 0.006s
I20260929 10:28:04.032066 21609 heartbeater.cc:503] Master 127.20.92.62:45475 was elected leader, sending a full tablet report...
I20260929 10:28:04.032060 21895 consensus_queue.cc:260] T 4bb305fb28384540863cdce02aed60dc P 17f48f89663e4bc8a1d48bf8e16edd41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d71ab203d844ab2b31248796b67d76e" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 33733 } } peers { permanent_uuid: "17f48f89663e4bc8a1d48bf8e16edd41" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38319 } } peers { permanent_uuid: "fb9db56f0cc34c7cb653306f3dac1f05" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39489 } }
I20260929 10:28:04.032840 21877 heartbeater.cc:503] Master 127.20.92.62:45475 was elected leader, sending a full tablet report...
I20260929 10:28:04.034785 21895 ts_tablet_manager.cc:1428] T 4bb305fb28384540863cdce02aed60dc P 17f48f89663e4bc8a1d48bf8e16edd41: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260929 10:28:04.103894 21900 raft_consensus.cc:482] T 4bb305fb28384540863cdce02aed60dc P 4d71ab203d844ab2b31248796b67d76e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:28:04.104102 21900 raft_consensus.cc:504] T 4bb305fb28384540863cdce02aed60dc P 4d71ab203d844ab2b31248796b67d76e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d71ab203d844ab2b31248796b67d76e" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 33733 } } peers { permanent_uuid: "17f48f89663e4bc8a1d48bf8e16edd41" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38319 } } peers { permanent_uuid: "fb9db56f0cc34c7cb653306f3dac1f05" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39489 } }
I20260929 10:28:04.105388 21900 leader_election.cc:288] T 4bb305fb28384540863cdce02aed60dc P 4d71ab203d844ab2b31248796b67d76e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17f48f89663e4bc8a1d48bf8e16edd41 (127.20.92.3:38319), fb9db56f0cc34c7cb653306f3dac1f05 (127.20.92.1:39489)
I20260929 10:28:04.109292 21891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:04.105772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:59277 (local address 127.20.92.3:38319)
0929 10:28:04.106074 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:04.106079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:04.106097 (+ 18us) server_negotiation.cc:407] Connection header received
0929 10:28:04.106255 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:04.106258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:04.106303 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:04.106450 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:04.106751 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:04.107260 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:04.108337 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:04.108733 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:04.108855 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:04.108931 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:04.109101 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 10:28:04.109167 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260929 10:28:04.109292 21745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:04.105621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:38319 (local address 127.20.92.2:59277)
0929 10:28:04.105872 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:04.105894 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:04.106008 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:04.106479 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:04.106482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:04.106499 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:04.106636 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:04.106642 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:04.107387 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:04.107390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:04.108206 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:04.108214 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:04.108858 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:04.108862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:04.108967 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:04.108988 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 10:28:04.109086 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 10:28:04.109157 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 10:28:04.109815 21893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:04.105973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:53401 (local address 127.20.92.1:39489)
0929 10:28:04.106128 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:04.106136 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:04.106376 (+ 240us) server_negotiation.cc:407] Connection header received
0929 10:28:04.106503 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:04.106509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:04.106566 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:04.106668 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:04.106974 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:04.107540 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:04.108665 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:04.109123 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:04.109265 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:04.109327 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:04.109559 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 10:28:04.109647 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 10:28:04.110176 21903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:04.105874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:39489 (local address 127.20.92.2:53401)
0929 10:28:04.106276 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:04.106287 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:04.106403 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:04.106678 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:04.106696 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:04.106713 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:04.106851 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:04.106857 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:04.107666 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:04.107673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:04.108529 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:04.108537 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:04.109264 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:04.109268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:04.109385 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:04.109416 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 10:28:04.109995 (+ 579us) client_negotiation.cc:240] Negotiation successful
0929 10:28:04.110070 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":134,"threads_started":1}
I20260929 10:28:04.110133 21831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb305fb28384540863cdce02aed60dc" candidate_uuid: "4d71ab203d844ab2b31248796b67d76e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17f48f89663e4bc8a1d48bf8e16edd41" is_pre_election: true
I20260929 10:28:04.110507 21831 raft_consensus.cc:2454] T 4bb305fb28384540863cdce02aed60dc P 17f48f89663e4bc8a1d48bf8e16edd41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d71ab203d844ab2b31248796b67d76e in term 0.
I20260929 10:28:04.111101 21632 leader_election.cc:302] T 4bb305fb28384540863cdce02aed60dc P 4d71ab203d844ab2b31248796b67d76e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17f48f89663e4bc8a1d48bf8e16edd41, 4d71ab203d844ab2b31248796b67d76e; no voters:
I20260929 10:28:04.111117 21563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb305fb28384540863cdce02aed60dc" candidate_uuid: "4d71ab203d844ab2b31248796b67d76e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb9db56f0cc34c7cb653306f3dac1f05" is_pre_election: true
I20260929 10:28:04.111461 21563 raft_consensus.cc:2454] T 4bb305fb28384540863cdce02aed60dc P fb9db56f0cc34c7cb653306f3dac1f05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d71ab203d844ab2b31248796b67d76e in term 0.
I20260929 10:28:04.111502 21900 raft_consensus.cc:2781] T 4bb305fb28384540863cdce02aed60dc P 4d71ab203d844ab2b31248796b67d76e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:28:04.111589 21900 raft_consensus.cc:482] T 4bb305fb28384540863cdce02aed60dc P 4d71ab203d844ab2b31248796b67d76e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:28:04.111644 21900 raft_consensus.cc:3037] T 4bb305fb28384540863cdce02aed60dc P 4d71ab203d844ab2b31248796b67d76e [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:04.112843 21900 raft_consensus.cc:504] T 4bb305fb28384540863cdce02aed60dc P 4d71ab203d844ab2b31248796b67d76e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d71ab203d844ab2b31248796b67d76e" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 33733 } } peers { permanent_uuid: "17f48f89663e4bc8a1d48bf8e16edd41" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38319 } } peers { permanent_uuid: "fb9db56f0cc34c7cb653306f3dac1f05" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39489 } }
I20260929 10:28:04.113189 21900 leader_election.cc:288] T 4bb305fb28384540863cdce02aed60dc P 4d71ab203d844ab2b31248796b67d76e [CANDIDATE]: Term 1 election: Requested vote from peers 17f48f89663e4bc8a1d48bf8e16edd41 (127.20.92.3:38319), fb9db56f0cc34c7cb653306f3dac1f05 (127.20.92.1:39489)
I20260929 10:28:04.113583 21563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb305fb28384540863cdce02aed60dc" candidate_uuid: "4d71ab203d844ab2b31248796b67d76e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb9db56f0cc34c7cb653306f3dac1f05"
I20260929 10:28:04.113579 21831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb305fb28384540863cdce02aed60dc" candidate_uuid: "4d71ab203d844ab2b31248796b67d76e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17f48f89663e4bc8a1d48bf8e16edd41"
I20260929 10:28:04.113734 21563 raft_consensus.cc:3037] T 4bb305fb28384540863cdce02aed60dc P fb9db56f0cc34c7cb653306f3dac1f05 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:04.113749 21831 raft_consensus.cc:3037] T 4bb305fb28384540863cdce02aed60dc P 17f48f89663e4bc8a1d48bf8e16edd41 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:04.115031 21831 raft_consensus.cc:2454] T 4bb305fb28384540863cdce02aed60dc P 17f48f89663e4bc8a1d48bf8e16edd41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d71ab203d844ab2b31248796b67d76e in term 1.
I20260929 10:28:04.115031 21563 raft_consensus.cc:2454] T 4bb305fb28384540863cdce02aed60dc P fb9db56f0cc34c7cb653306f3dac1f05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d71ab203d844ab2b31248796b67d76e in term 1.
I20260929 10:28:04.115525 21633 leader_election.cc:302] T 4bb305fb28384540863cdce02aed60dc P 4d71ab203d844ab2b31248796b67d76e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d71ab203d844ab2b31248796b67d76e, fb9db56f0cc34c7cb653306f3dac1f05; no voters:
I20260929 10:28:04.115836 21900 raft_consensus.cc:2781] T 4bb305fb28384540863cdce02aed60dc P 4d71ab203d844ab2b31248796b67d76e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:04.116087 21900 raft_consensus.cc:686] T 4bb305fb28384540863cdce02aed60dc P 4d71ab203d844ab2b31248796b67d76e [term 1 LEADER]: Becoming Leader. State: Replica: 4d71ab203d844ab2b31248796b67d76e, State: Running, Role: LEADER
I20260929 10:28:04.116452 21900 consensus_queue.cc:237] T 4bb305fb28384540863cdce02aed60dc P 4d71ab203d844ab2b31248796b67d76e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d71ab203d844ab2b31248796b67d76e" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 33733 } } peers { permanent_uuid: "17f48f89663e4bc8a1d48bf8e16edd41" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38319 } } peers { permanent_uuid: "fb9db56f0cc34c7cb653306f3dac1f05" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39489 } }
I20260929 10:28:04.119829 21417 catalog_manager.cc:5495] T 4bb305fb28384540863cdce02aed60dc P 4d71ab203d844ab2b31248796b67d76e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d71ab203d844ab2b31248796b67d76e (127.20.92.2). New cstate: current_term: 1 leader_uuid: "4d71ab203d844ab2b31248796b67d76e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d71ab203d844ab2b31248796b67d76e" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 33733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17f48f89663e4bc8a1d48bf8e16edd41" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb9db56f0cc34c7cb653306f3dac1f05" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39489 } health_report { overall_health: UNKNOWN } } }
Sep 29 10:28:04 dist-test-slave-9cj5 krb5kdc[21373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.1@KRBTEST.COM
I20260929 10:28:04.163096 21893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:04.153042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46386 (local address 127.20.92.1:39489)
0929 10:28:04.153205 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:04.153210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:04.153368 (+ 158us) server_negotiation.cc:407] Connection header received
0929 10:28:04.153549 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:04.153557 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:04.153601 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:04.153692 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:04.154521 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:04.155096 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:04.155982 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:04.156225 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:04.156337 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:04.159062 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:04.159075 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:04.159092 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:04.159123 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:04.161051 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:04.161573 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:04.161579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:04.161580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:04.161668 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:04.162053 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:04.162056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:04.162058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:04.162334 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:04.162567 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:04.162783 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 10:28:04.162927 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
Sep 29 10:28:04 dist-test-slave-9cj5 krb5kdc[21373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677684, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260929 10:28:04.180974 21878 tablet.cc:2369] T 4bb305fb28384540863cdce02aed60dc P 17f48f89663e4bc8a1d48bf8e16edd41: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 10:28:04 dist-test-slave-9cj5 krb5kdc[21373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677684, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.92.1@KRBTEST.COM
I20260929 10:28:04.194000 21893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:04.183542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46392 (local address 127.20.92.1:39489)
0929 10:28:04.183756 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:04.183761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:04.183933 (+ 172us) server_negotiation.cc:407] Connection header received
0929 10:28:04.184128 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:04.184137 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:04.184184 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:04.184287 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:04.185163 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:04.185838 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:04.186904 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:04.187231 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:04.187387 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:04.190364 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:04.190369 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:04.190372 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:04.190416 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:04.192052 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:04.192554 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:04.192558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:04.192560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:04.192657 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:04.192964 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:04.192967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:04.192969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:04.193164 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:04.193305 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:04.193639 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 10:28:04.193819 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260929 10:28:04.201037 21610 tablet.cc:2369] T 4bb305fb28384540863cdce02aed60dc P fb9db56f0cc34c7cb653306f3dac1f05: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:28:04.202545 21744 tablet.cc:2369] T 4bb305fb28384540863cdce02aed60dc P 4d71ab203d844ab2b31248796b67d76e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 10:28:04 dist-test-slave-9cj5 krb5kdc[21373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 10:28:04 dist-test-slave-9cj5 krb5kdc[21373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.1@KRBTEST.COM
I20260929 10:28:04.223459 21893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:04.212446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46396 (local address 127.20.92.1:39489)
0929 10:28:04.212638 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:04.212641 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:04.212661 (+ 20us) server_negotiation.cc:407] Connection header received
0929 10:28:04.212907 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:04.212910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:04.212960 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:04.213039 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:04.214095 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:04.214612 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:04.215695 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:04.216000 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:04.216292 (+ 292us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:04.219404 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:04.219408 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:04.219410 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:04.219446 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:04.221181 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:04.221924 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:04.221928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:04.221929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:04.222023 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:04.222428 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:04.222431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:04.222432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:04.222644 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:04.222769 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:04.223190 (+ 421us) server_negotiation.cc:299] Negotiation successful
0929 10:28:04.223313 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260929 10:28:04.225165 21523 tablet_service.cc:2908] Rejecting scan request for tablet 4bb305fb28384540863cdce02aed60dc: Uninitialized: clean time has not yet been initialized
I20260929 10:28:04.227766 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 21480
I20260929 10:28:04.233908 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 21614
I20260929 10:28:04.240274 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 21748
I20260929 10:28:04.246255 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 21386
2026-09-29T10:28:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2587 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:28:04 dist-test-slave-9cj5 krb5kdc[21918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:28:04 dist-test-slave-9cj5 krb5kdc[21918](info): set up 2 sockets
Sep 29 10:28:04 dist-test-slave-9cj5 krb5kdc[21918](info): commencing operation
krb5kdc: starting...
W20260929 10:28:05.350665 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:28:05 dist-test-slave-9cj5 krb5kdc[21918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:28:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:28:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
I20260929 10:28:05.482295 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:34829
--webserver_interface=127.20.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:34829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:05.603181 21931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:05.603559 21931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:05.603649 21931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:05.607894 21931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:28:05.608017 21931 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 10:28:05.608049 21931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:05.608093 21931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:05.608150 21931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:28:05.608187 21931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:28:05.613768 21931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:34829
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:34829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:05.614948 21931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:05.616015 21931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:05.624810 21939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:05.624833 21936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:05.624814 21937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:05.625677 21931 server_base.cc:1034] running on GCE node
I20260929 10:28:05.626549 21931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:05.627810 21931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:05.629048 21931 hybrid_clock.cc:648] HybridClock initialized: now 1790677685629031 us; error 34 us; skew 500 ppm
Sep 29 10:28:05 dist-test-slave-9cj5 krb5kdc[21918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677685, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:05.632346 21931 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:28:05.633378 21931 webserver.cc:460] Webserver started at http://127.20.92.62:39305/ using document root <none> and password file <none>
I20260929 10:28:05.634027 21931 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:05.634091 21931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:05.634318 21931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:05.636502 21931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "70ba2352c99b489d95f2c7ae38dca8b2"
format_stamp: "Formatted at 2026-09-29 10:28:05 on dist-test-slave-9cj5"
server_key: "008f14950db2dda565167c9d00fa6f70"
server_key_iv: "cca60b4a1ab0e58a6b463a0b3447bf2c"
server_key_version: "encryptionkey@0"
I20260929 10:28:05.637108 21931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "70ba2352c99b489d95f2c7ae38dca8b2"
format_stamp: "Formatted at 2026-09-29 10:28:05 on dist-test-slave-9cj5"
server_key: "008f14950db2dda565167c9d00fa6f70"
server_key_iv: "cca60b4a1ab0e58a6b463a0b3447bf2c"
server_key_version: "encryptionkey@0"
I20260929 10:28:05.640995 21931 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 10:28:05.643546 21946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:05.644886 21931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:28:05.645032 21931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "70ba2352c99b489d95f2c7ae38dca8b2"
format_stamp: "Formatted at 2026-09-29 10:28:05 on dist-test-slave-9cj5"
server_key: "008f14950db2dda565167c9d00fa6f70"
server_key_iv: "cca60b4a1ab0e58a6b463a0b3447bf2c"
server_key_version: "encryptionkey@0"
I20260929 10:28:05.645149 21931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:05.659009 21931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:05.662475 21931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:05.662714 21931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:05.671440 21931 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:34829
I20260929 10:28:05.671466 22008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:34829 every 8 connection(s)
I20260929 10:28:05.672490 21931 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:28:05.675437 22009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:05.679513 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 21931
I20260929 10:28:05.679618 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:28:05.679930 20848 external_mini_cluster.cc:1428] Setting key 2aa53ebf2798f78f4f3c56b72ad0455a
I20260929 10:28:05.682619 22009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70ba2352c99b489d95f2c7ae38dca8b2: Bootstrap starting.
I20260929 10:28:05.685488 22009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70ba2352c99b489d95f2c7ae38dca8b2: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:05.686425 22009 log.cc:824] T 00000000000000000000000000000000 P 70ba2352c99b489d95f2c7ae38dca8b2: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:05.689191 22009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70ba2352c99b489d95f2c7ae38dca8b2: No bootstrap required, opened a new log
Sep 29 10:28:05 dist-test-slave-9cj5 krb5kdc[21918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:05.693385 22009 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70ba2352c99b489d95f2c7ae38dca8b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ba2352c99b489d95f2c7ae38dca8b2" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34829 } }
I20260929 10:28:05.693686 22009 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70ba2352c99b489d95f2c7ae38dca8b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:05.693795 22009 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70ba2352c99b489d95f2c7ae38dca8b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70ba2352c99b489d95f2c7ae38dca8b2, State: Initialized, Role: FOLLOWER
I20260929 10:28:05.694514 22009 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70ba2352c99b489d95f2c7ae38dca8b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ba2352c99b489d95f2c7ae38dca8b2" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34829 } }
I20260929 10:28:05.694739 22009 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70ba2352c99b489d95f2c7ae38dca8b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:28:05.694839 22009 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70ba2352c99b489d95f2c7ae38dca8b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:28:05.694983 22009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70ba2352c99b489d95f2c7ae38dca8b2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:05.696257 22009 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70ba2352c99b489d95f2c7ae38dca8b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ba2352c99b489d95f2c7ae38dca8b2" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34829 } }
I20260929 10:28:05.696645 22009 leader_election.cc:302] T 00000000000000000000000000000000 P 70ba2352c99b489d95f2c7ae38dca8b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70ba2352c99b489d95f2c7ae38dca8b2; no voters:
I20260929 10:28:05.697019 22009 leader_election.cc:288] T 00000000000000000000000000000000 P 70ba2352c99b489d95f2c7ae38dca8b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:28:05.697149 22014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70ba2352c99b489d95f2c7ae38dca8b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:05.697365 22014 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70ba2352c99b489d95f2c7ae38dca8b2 [term 1 LEADER]: Becoming Leader. State: Replica: 70ba2352c99b489d95f2c7ae38dca8b2, State: Running, Role: LEADER
I20260929 10:28:05.697369 22012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:05.681591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36908 (local address 127.20.92.62:34829)
0929 10:28:05.682112 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:05.682128 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:05.682165 (+ 37us) server_negotiation.cc:407] Connection header received
0929 10:28:05.683304 (+ 1139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:05.683325 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:05.683719 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:05.684033 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:05.685106 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:05.686325 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:05.687496 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:05.688038 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:05.688237 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:05.691277 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:05.691283 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:05.691301 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:05.691335 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:05.693353 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:05.693982 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:05.693993 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:05.693998 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:05.694124 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:05.694609 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:05.694612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:05.694614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:05.695285 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:05.695471 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:05.695841 (+ 370us) server_negotiation.cc:299] Negotiation successful
0929 10:28:05.696237 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":204,"threads_started":1}
I20260929 10:28:05.697738 22014 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70ba2352c99b489d95f2c7ae38dca8b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ba2352c99b489d95f2c7ae38dca8b2" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34829 } }
I20260929 10:28:05.698402 22009 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70ba2352c99b489d95f2c7ae38dca8b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:28:05.699062 22014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70ba2352c99b489d95f2c7ae38dca8b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70ba2352c99b489d95f2c7ae38dca8b2. Latest consensus state: current_term: 1 leader_uuid: "70ba2352c99b489d95f2c7ae38dca8b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ba2352c99b489d95f2c7ae38dca8b2" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34829 } } }
I20260929 10:28:05.699177 22014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70ba2352c99b489d95f2c7ae38dca8b2 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:05.699493 22016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70ba2352c99b489d95f2c7ae38dca8b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70ba2352c99b489d95f2c7ae38dca8b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ba2352c99b489d95f2c7ae38dca8b2" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34829 } } }
I20260929 10:28:05.699595 22016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70ba2352c99b489d95f2c7ae38dca8b2 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:05.700134 22020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:28:05.703253 22020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:28:05.708871 22020 catalog_manager.cc:1347] Generated new cluster ID: 65e63e6ea65a4374941b5653b3ef852a
I20260929 10:28:05.709017 22020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:28:05.738180 22020 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:28:05.739320 22020 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:28:05.745779 22020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70ba2352c99b489d95f2c7ae38dca8b2: Generated new TSK 0
I20260929 10:28:05.746694 22020 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
I20260929 10:28:05.777652 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:0
--local_ip_for_outbound_sockets=127.20.92.1
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:34829
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:05.905205 22035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:05.905515 22035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:05.905565 22035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:05.909709 22035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:05.909855 22035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:05.909917 22035 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 10:28:05.909955 22035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:28:05.915572 22035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.92.62:34829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:05.916627 22035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:05.917670 22035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:05.928170 22041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:05.928347 22035 server_base.cc:1034] running on GCE node
W20260929 10:28:05.928151 22040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:05.928138 22043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:05.929060 22035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:05.929932 22035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:05.931134 22035 hybrid_clock.cc:648] HybridClock initialized: now 1790677685931113 us; error 35 us; skew 500 ppm
Sep 29 10:28:05 dist-test-slave-9cj5 krb5kdc[21918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677685, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:05.934983 22035 init.cc:377] Logged in from keytab as kudu/127.20.92.1@KRBTEST.COM (short username kudu)
I20260929 10:28:05.936093 22035 webserver.cc:460] Webserver started at http://127.20.92.1:37237/ using document root <none> and password file <none>
I20260929 10:28:05.936784 22035 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:05.936902 22035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:05.937134 22035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:05.939432 22035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-0/data/instance:
uuid: "7e91d274147b4563940974d85c4b34d5"
format_stamp: "Formatted at 2026-09-29 10:28:05 on dist-test-slave-9cj5"
server_key: "f39c01082ffda9b7055856d66df1df3d"
server_key_iv: "4415690bd6b276c0d23ccfd06d156c35"
server_key_version: "encryptionkey@0"
I20260929 10:28:05.940040 22035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance:
uuid: "7e91d274147b4563940974d85c4b34d5"
format_stamp: "Formatted at 2026-09-29 10:28:05 on dist-test-slave-9cj5"
server_key: "f39c01082ffda9b7055856d66df1df3d"
server_key_iv: "4415690bd6b276c0d23ccfd06d156c35"
server_key_version: "encryptionkey@0"
I20260929 10:28:05.944198 22035 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:28:05.947121 22050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:05.948597 22035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:28:05.948740 22035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "7e91d274147b4563940974d85c4b34d5"
format_stamp: "Formatted at 2026-09-29 10:28:05 on dist-test-slave-9cj5"
server_key: "f39c01082ffda9b7055856d66df1df3d"
server_key_iv: "4415690bd6b276c0d23ccfd06d156c35"
server_key_version: "encryptionkey@0"
I20260929 10:28:05.948868 22035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:05.970429 22035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:05.973701 22035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:05.973999 22035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:05.975402 22035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:05.975453 22035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:05.975530 22035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:05.975564 22035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 10:28:05 dist-test-slave-9cj5 krb5kdc[21918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677685, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:05.988497 22035 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:36413
I20260929 10:28:05.988556 22165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:36413 every 8 connection(s)
I20260929 10:28:05.989524 22035 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
I20260929 10:28:05.990614 22012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:05.976490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:50155 (local address 127.20.92.62:34829)
0929 10:28:05.976621 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:05.976625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:05.977713 (+ 1088us) server_negotiation.cc:407] Connection header received
0929 10:28:05.978913 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:05.978917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:05.978985 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:05.979084 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:05.980345 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:05.980905 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:05.982439 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:05.982685 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:05.982808 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:05.986433 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:05.986436 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:05.986438 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:05.986466 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:05.988193 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:05.988925 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:05.988928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:05.988930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:05.989016 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:05.989384 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:05.989387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:05.989389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:05.989592 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:05.989703 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:05.990326 (+ 623us) server_negotiation.cc:299] Negotiation successful
0929 10:28:05.990479 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 10:28:05.991830 22059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:05.977038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:34829 (local address 127.20.92.1:50155)
0929 10:28:05.977561 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:05.977604 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:05.978657 (+ 1053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:05.979258 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:05.979265 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:05.979685 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:05.980190 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:05.980203 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:05.981040 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:05.981050 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:05.982292 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:05.982303 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:05.982860 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:05.982864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:05.983097 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:05.983776 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:05.983796 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:05.986217 (+ 2421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:05.988330 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:05.988336 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:05.988353 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:05.988798 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:05.989117 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:05.989120 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:05.989122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:05.989269 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:05.989700 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:05.989707 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:05.990022 (+ 315us) client_negotiation.cc:769] Sending connection context
0929 10:28:05.990798 (+ 776us) client_negotiation.cc:240] Negotiation successful
0929 10:28:05.991088 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":143,"threads_started":1}
I20260929 10:28:05.993361 22166 heartbeater.cc:347] Connected to a master server at 127.20.92.62:34829
I20260929 10:28:05.993626 22166 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:05.994246 22166 heartbeater.cc:511] Master 127.20.92.62:34829 requested a full tablet report, sending...
I20260929 10:28:05.995502 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 22035
I20260929 10:28:05.995616 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance
I20260929 10:28:05.995908 20848 external_mini_cluster.cc:1428] Setting key d9b62b2205d7839d2f727cfc47dbf517
I20260929 10:28:05.996142 21963 ts_manager.cc:194] Registered new tserver with Master: 7e91d274147b4563940974d85c4b34d5 (127.20.92.1:36413)
I20260929 10:28:05.997589 21963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.1@KRBTEST.COM'} at 127.20.92.1:50155
I20260929 10:28:06.004683 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:05.996071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:34829 (local address 127.20.92.1:42545)
0929 10:28:05.996318 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:05.996340 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:05.996465 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:05.996714 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:05.996717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:05.996933 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:05.997035 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:05.997040 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:05.998048 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:05.998052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:05.998841 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:05.998855 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:05.999526 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:05.999530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:05.999622 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:06.000259 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:06.000274 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:06.000752 (+ 478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:06.002987 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:06.002990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:06.002991 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:06.003194 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:06.003527 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:06.003529 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:06.003531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:06.003604 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:06.004139 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:06.004142 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:06.004251 (+ 109us) client_negotiation.cc:769] Sending connection context
0929 10:28:06.004364 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 10:28:06.004501 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":97,"threads_started":1}
I20260929 10:28:06.004781 22012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:05.995859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:42545 (local address 127.20.92.62:34829)
0929 10:28:05.995989 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:05.995995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:05.996416 (+ 421us) server_negotiation.cc:407] Connection header received
0929 10:28:05.996561 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:05.996564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:05.996608 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:05.996699 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:05.997151 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:05.997879 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:05.998976 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:05.999394 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:05.999516 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:06.000912 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:06.000918 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:06.000925 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:06.000956 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:06.002844 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:06.003299 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:06.003306 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:06.003310 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:06.003421 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:06.003700 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:06.003705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:06.003709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:06.003990 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:06.004104 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:06.004453 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 10:28:06.004594 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
WARNING: no policy specified for kudu/127.20.92.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
I20260929 10:28:06.031064 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:0
--local_ip_for_outbound_sockets=127.20.92.2
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:34829
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:06.150400 22175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:06.150736 22175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:06.150831 22175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:06.155269 22175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:06.155376 22175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:06.155447 22175 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 10:28:06.155535 22175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:28:06.161907 22175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.92.62:34829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:06.163123 22175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:06.164203 22175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:06.174367 22181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:06.174347 22180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:06.174347 22183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:06.174655 22175 server_base.cc:1034] running on GCE node
I20260929 10:28:06.175148 22175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:06.175981 22175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:06.177163 22175 hybrid_clock.cc:648] HybridClock initialized: now 1790677686177147 us; error 31 us; skew 500 ppm
Sep 29 10:28:06 dist-test-slave-9cj5 krb5kdc[21918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677686, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:06.180258 22175 init.cc:377] Logged in from keytab as kudu/127.20.92.2@KRBTEST.COM (short username kudu)
I20260929 10:28:06.181298 22175 webserver.cc:460] Webserver started at http://127.20.92.2:33033/ using document root <none> and password file <none>
I20260929 10:28:06.182026 22175 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:06.182124 22175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:06.182758 22175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:06.184976 22175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-1/data/instance:
uuid: "fb040c004af747e9806f728b7eeb3022"
format_stamp: "Formatted at 2026-09-29 10:28:06 on dist-test-slave-9cj5"
server_key: "caa9eb2ddb16b5c5c61ef6b51090940c"
server_key_iv: "a890e8ee111ae68629e0a92878966ebc"
server_key_version: "encryptionkey@0"
I20260929 10:28:06.185580 22175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance:
uuid: "fb040c004af747e9806f728b7eeb3022"
format_stamp: "Formatted at 2026-09-29 10:28:06 on dist-test-slave-9cj5"
server_key: "caa9eb2ddb16b5c5c61ef6b51090940c"
server_key_iv: "a890e8ee111ae68629e0a92878966ebc"
server_key_version: "encryptionkey@0"
I20260929 10:28:06.190490 22175 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 10:28:06.193501 22190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:06.195106 22175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:28:06.195276 22175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "fb040c004af747e9806f728b7eeb3022"
format_stamp: "Formatted at 2026-09-29 10:28:06 on dist-test-slave-9cj5"
server_key: "caa9eb2ddb16b5c5c61ef6b51090940c"
server_key_iv: "a890e8ee111ae68629e0a92878966ebc"
server_key_version: "encryptionkey@0"
I20260929 10:28:06.195439 22175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:06.218155 22175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:06.221819 22175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:06.222038 22175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:06.223820 22175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:06.223927 22175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:06.223999 22175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:06.224053 22175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 10:28:06 dist-test-slave-9cj5 krb5kdc[21918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677686, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:06.236793 22175 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:37381
I20260929 10:28:06.236847 22305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:37381 every 8 connection(s)
I20260929 10:28:06.238071 22175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
I20260929 10:28:06.238946 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 22175
I20260929 10:28:06.239135 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance
I20260929 10:28:06.239521 20848 external_mini_cluster.cc:1428] Setting key e083c107f13c9fefec34dc9f3ababe26
I20260929 10:28:06.241953 22012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:06.224979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:45207 (local address 127.20.92.62:34829)
0929 10:28:06.225118 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:06.225122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:06.226333 (+ 1211us) server_negotiation.cc:407] Connection header received
0929 10:28:06.227786 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:06.227790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:06.227933 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:06.228049 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:06.229879 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.230505 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.232123 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.232382 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.232515 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:06.236590 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:06.236597 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:06.236602 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:06.236643 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:06.239150 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:06.239871 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:06.239877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:06.239879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:06.239996 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:06.240509 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:06.240512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:06.240514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:06.240716 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:06.240850 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:06.241543 (+ 693us) server_negotiation.cc:299] Negotiation successful
0929 10:28:06.241670 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 10:28:06.243201 22199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:06.225554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:34829 (local address 127.20.92.2:45207)
0929 10:28:06.226131 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:06.226191 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:06.227500 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:06.228317 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:06.228331 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:06.228902 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:06.229654 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:06.229683 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.230666 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.230670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:06.231957 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:06.231969 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.232583 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.232587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:06.232794 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:06.233688 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:06.233705 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:06.236297 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:06.239299 (+ 3002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:06.239304 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:06.239316 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:06.239710 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:06.240099 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:06.240106 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:06.240109 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:06.240322 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:06.240880 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:06.240887 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:06.241221 (+ 334us) client_negotiation.cc:769] Sending connection context
0929 10:28:06.241459 (+ 238us) client_negotiation.cc:240] Negotiation successful
0929 10:28:06.241897 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":176,"threads_started":1}
I20260929 10:28:06.245503 22306 heartbeater.cc:347] Connected to a master server at 127.20.92.62:34829
I20260929 10:28:06.245682 22306 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:06.246446 22306 heartbeater.cc:511] Master 127.20.92.62:34829 requested a full tablet report, sending...
I20260929 10:28:06.247820 21962 ts_manager.cc:194] Registered new tserver with Master: fb040c004af747e9806f728b7eeb3022 (127.20.92.2:37381)
I20260929 10:28:06.248616 21962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.2@KRBTEST.COM'} at 127.20.92.2:45207
WARNING: no policy specified for kudu/127.20.92.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.3@KRBTEST.COM" created.
I20260929 10:28:06.259533 22012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:06.248137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:48849 (local address 127.20.92.62:34829)
0929 10:28:06.248259 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:06.248264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:06.249207 (+ 943us) server_negotiation.cc:407] Connection header received
0929 10:28:06.251088 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:06.251092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:06.251151 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:06.251256 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:06.251816 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.252516 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.253641 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.254060 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.254188 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:06.255643 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:06.255646 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:06.255648 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:06.255674 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:06.257361 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:06.257954 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:06.257957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:06.257959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:06.258073 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:06.258470 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:06.258473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:06.258475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:06.258744 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:06.258885 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:06.259194 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 10:28:06.259326 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 10:28:06.259552 22313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:06.248072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:34829 (local address 127.20.92.2:48849)
0929 10:28:06.249080 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:06.249094 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:06.250948 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:06.251278 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:06.251281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:06.251507 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:06.251615 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:06.251622 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.252651 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.252655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:06.253396 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:06.253406 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.254203 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.254207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:06.254313 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:06.254866 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:06.254889 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:06.255454 (+ 565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:06.257504 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:06.257508 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:06.257509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:06.257740 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:06.258200 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:06.258206 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:06.258208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:06.258306 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:06.258870 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:06.258874 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:06.258998 (+ 124us) client_negotiation.cc:769] Sending connection context
0929 10:28:06.259120 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 10:28:06.259327 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
I20260929 10:28:06.275257 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:0
--local_ip_for_outbound_sockets=127.20.92.3
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:34829
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:06.398026 22315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:06.398706 22315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:06.398872 22315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:06.404342 22315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:06.404445 22315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:06.404500 22315 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 10:28:06.404541 22315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:28:06.410463 22315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.92.62:34829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:06.411660 22315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:06.412675 22315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:06.422935 22321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:06.423005 22323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:06.422935 22320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:06.423425 22315 server_base.cc:1034] running on GCE node
I20260929 10:28:06.423873 22315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:06.424654 22315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:06.425906 22315 hybrid_clock.cc:648] HybridClock initialized: now 1790677686425886 us; error 35 us; skew 500 ppm
Sep 29 10:28:06 dist-test-slave-9cj5 krb5kdc[21918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677686, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:06.428900 22315 init.cc:377] Logged in from keytab as kudu/127.20.92.3@KRBTEST.COM (short username kudu)
I20260929 10:28:06.430034 22315 webserver.cc:460] Webserver started at http://127.20.92.3:45319/ using document root <none> and password file <none>
I20260929 10:28:06.430591 22315 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:06.430655 22315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:06.430830 22315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:06.433097 22315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-2/data/instance:
uuid: "bb662e153eb5403ea049e77809020c45"
format_stamp: "Formatted at 2026-09-29 10:28:06 on dist-test-slave-9cj5"
server_key: "0ffed37677e884cbc5be6dcd1639b389"
server_key_iv: "5b6f1ffb01a224fc3f5790b4e4af7cb4"
server_key_version: "encryptionkey@0"
I20260929 10:28:06.433693 22315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance:
uuid: "bb662e153eb5403ea049e77809020c45"
format_stamp: "Formatted at 2026-09-29 10:28:06 on dist-test-slave-9cj5"
server_key: "0ffed37677e884cbc5be6dcd1639b389"
server_key_iv: "5b6f1ffb01a224fc3f5790b4e4af7cb4"
server_key_version: "encryptionkey@0"
I20260929 10:28:06.438295 22315 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 10:28:06.441860 22330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:06.443444 22315 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 10:28:06.443573 22315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "bb662e153eb5403ea049e77809020c45"
format_stamp: "Formatted at 2026-09-29 10:28:06 on dist-test-slave-9cj5"
server_key: "0ffed37677e884cbc5be6dcd1639b389"
server_key_iv: "5b6f1ffb01a224fc3f5790b4e4af7cb4"
server_key_version: "encryptionkey@0"
I20260929 10:28:06.443688 22315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:06.469000 22315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:06.472689 22315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:06.472927 22315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:06.474525 22315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:06.474604 22315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:06.474682 22315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:06.474726 22315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 10:28:06 dist-test-slave-9cj5 krb5kdc[21918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677686, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:06.487119 22315 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:40075
I20260929 10:28:06.487207 22445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:40075 every 8 connection(s)
I20260929 10:28:06.488294 22315 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:28:06.491782 22012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:06.475792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:51697 (local address 127.20.92.62:34829)
0929 10:28:06.475972 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:06.475978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:06.476851 (+ 873us) server_negotiation.cc:407] Connection header received
0929 10:28:06.478249 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:06.478253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:06.478325 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:06.478445 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:06.480055 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.480835 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.482747 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.483211 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.483366 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:06.486996 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:06.487004 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:06.487008 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:06.487050 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:06.489268 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:06.489951 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:06.489954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:06.489956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:06.490043 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:06.490456 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:06.490459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:06.490460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:06.490661 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:06.490785 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:06.491456 (+ 671us) server_negotiation.cc:299] Negotiation successful
0929 10:28:06.491585 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 10:28:06.492588 22339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:06.476208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:34829 (local address 127.20.92.3:51697)
0929 10:28:06.476687 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:06.476741 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:06.477953 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:06.478639 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:06.478646 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:06.479125 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:06.479859 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:06.479886 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.480975 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.480980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:06.482578 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:06.482593 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.483426 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.483431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:06.483630 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:06.484404 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:06.484424 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:06.486762 (+ 2338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:06.489448 (+ 2686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:06.489454 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:06.489467 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:06.489826 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:06.490166 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:06.490169 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:06.490170 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:06.490342 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:06.490945 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:06.490951 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:06.491189 (+ 238us) client_negotiation.cc:769] Sending connection context
0929 10:28:06.491347 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 10:28:06.491619 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":139,"threads_started":1}
I20260929 10:28:06.493261 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 22315
I20260929 10:28:06.493431 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance
I20260929 10:28:06.493772 20848 external_mini_cluster.cc:1428] Setting key 25d4f95c5dc2aee1ef9447e73c1399a3
I20260929 10:28:06.494289 22446 heartbeater.cc:347] Connected to a master server at 127.20.92.62:34829
I20260929 10:28:06.494537 22446 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:06.495180 22446 heartbeater.cc:511] Master 127.20.92.62:34829 requested a full tablet report, sending...
I20260929 10:28:06.496131 21963 ts_manager.cc:194] Registered new tserver with Master: bb662e153eb5403ea049e77809020c45 (127.20.92.3:40075)
I20260929 10:28:06.496691 21963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.3@KRBTEST.COM'} at 127.20.92.3:51697
I20260929 10:28:06.505618 22012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:06.496829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:34439 (local address 127.20.92.62:34829)
0929 10:28:06.496954 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:06.496959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:06.497108 (+ 149us) server_negotiation.cc:407] Connection header received
0929 10:28:06.497219 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:06.497223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:06.497274 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:06.497354 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:06.497891 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.498630 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.499873 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.500219 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.500320 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:06.502139 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:06.502142 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:06.502144 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:06.502170 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:06.503804 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:06.504318 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:06.504321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:06.504323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:06.504407 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:06.504733 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:06.504736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:06.504737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:06.504974 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:06.505110 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:06.505398 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 10:28:06.505502 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 10:28:06.505620 22452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:06.496691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:34829 (local address 127.20.92.3:34439)
0929 10:28:06.497017 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:06.497034 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:06.497134 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:06.497387 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:06.497390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:06.497622 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:06.497734 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:06.497778 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.498746 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.498750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:06.499744 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:06.499755 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.500362 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.500365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:06.500500 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:06.501475 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:06.501487 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:06.501972 (+ 485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:06.503930 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:06.503940 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:06.503942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:06.504163 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:06.504531 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:06.504534 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:06.504536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:06.504630 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:06.505120 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:06.505122 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:06.505215 (+ 93us) client_negotiation.cc:769] Sending connection context
0929 10:28:06.505325 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 10:28:06.505454 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":133,"threads_started":1}
I20260929 10:28:06.507692 20848 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 10:28:06 dist-test-slave-9cj5 krb5kdc[21918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677686, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 10:28:06 dist-test-slave-9cj5 krb5kdc[21918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677686, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:06.534946 22012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:06.525181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36922 (local address 127.20.92.62:34829)
0929 10:28:06.525380 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:06.525384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:06.525398 (+ 14us) server_negotiation.cc:407] Connection header received
0929 10:28:06.525440 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:06.525442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:06.525481 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:06.525554 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:06.526386 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.526987 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.527892 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.528113 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.528199 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:06.531029 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:06.531032 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:06.531034 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:06.531062 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:06.532743 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:06.533258 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:06.533261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:06.533262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:06.533355 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:06.533668 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:06.533671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:06.533673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:06.534090 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:06.534256 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:06.534659 (+ 403us) server_negotiation.cc:299] Negotiation successful
0929 10:28:06.534780 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 10:28:06.538292 21962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:28:06.541358 21962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:28:06.558038 22464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:06.551257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:40075 (local address 127.0.0.1:52982)
0929 10:28:06.551749 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:06.551804 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:06.551939 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:06.552737 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:06.552748 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:06.552781 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:06.552947 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:06.552957 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.554819 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.554823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:06.556445 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:06.556456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.557408 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.557411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:06.557524 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:06.557564 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 10:28:06.557685 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 10:28:06.557851 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":38,"thread_start_us":95,"threads_started":1}
I20260929 10:28:06.558245 22463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:06.552099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:37381 (local address 127.0.0.1:49806)
0929 10:28:06.553181 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:06.553194 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:06.553356 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:06.553949 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:06.553953 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:06.553973 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:06.554064 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:06.554071 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.555344 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.555348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:06.556659 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:06.556667 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.557693 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.557697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:06.557836 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:06.557945 (+ 109us) client_negotiation.cc:769] Sending connection context
0929 10:28:06.558022 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 10:28:06.558107 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1013,"mutex_wait_us":24,"thread_start_us":126,"threads_started":1}
I20260929 10:28:06.558249 22467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:06.551316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:36413 (local address 127.0.0.1:58608)
0929 10:28:06.552377 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:06.552391 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:06.552544 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:06.553182 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:06.553184 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:06.553207 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:06.553303 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:06.553311 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.554919 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.554923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:06.556382 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:06.556392 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.557660 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.557665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:06.557831 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:06.557876 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 10:28:06.557981 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 10:28:06.558064 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"mutex_wait_us":142,"thread_start_us":123,"threads_started":1}
I20260929 10:28:06.558717 22465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:06.551392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52982 (local address 127.20.92.3:40075)
0929 10:28:06.552109 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:06.552123 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:06.552151 (+ 28us) server_negotiation.cc:407] Connection header received
0929 10:28:06.552259 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:06.552265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:06.552505 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:06.553125 (+ 620us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:06.553327 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.554678 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.556600 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.557263 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.557399 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:06.557587 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:06.558478 (+ 891us) server_negotiation.cc:299] Negotiation successful
0929 10:28:06.558574 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":152,"threads_started":1}
I20260929 10:28:06.558717 22468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:06.552612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49806 (local address 127.20.92.2:37381)
0929 10:28:06.553109 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:06.553122 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:06.553286 (+ 164us) server_negotiation.cc:407] Connection header received
0929 10:28:06.553515 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:06.553520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:06.553712 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:06.553996 (+ 284us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:06.554172 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.555006 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.556788 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.557581 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.557691 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:06.557873 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:06.558478 (+ 605us) server_negotiation.cc:299] Negotiation successful
0929 10:28:06.558574 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":143,"threads_started":1}
I20260929 10:28:06.558995 22466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:06.551961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58608 (local address 127.20.92.1:36413)
0929 10:28:06.552216 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:06.552223 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:06.552474 (+ 251us) server_negotiation.cc:407] Connection header received
0929 10:28:06.552692 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:06.552697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:06.552880 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:06.552999 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:06.553420 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.554773 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.556513 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.557331 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.557444 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:06.557589 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:06.558020 (+ 431us) server_negotiation.cc:299] Negotiation successful
0929 10:28:06.558131 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":96,"threads_started":1}
I20260929 10:28:06.561539 22100 tablet_service.cc:1467] Processing CreateTablet for tablet eb942ebaa8a3496e8bc5d0e24fbe3cf7 (DEFAULT_TABLE table=test-table [id=0c7a5f85eae548cdb8b5791908648832]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:06.561630 22380 tablet_service.cc:1467] Processing CreateTablet for tablet eb942ebaa8a3496e8bc5d0e24fbe3cf7 (DEFAULT_TABLE table=test-table [id=0c7a5f85eae548cdb8b5791908648832]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:06.561630 22240 tablet_service.cc:1467] Processing CreateTablet for tablet eb942ebaa8a3496e8bc5d0e24fbe3cf7 (DEFAULT_TABLE table=test-table [id=0c7a5f85eae548cdb8b5791908648832]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:06.562664 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb942ebaa8a3496e8bc5d0e24fbe3cf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:06.562702 22380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb942ebaa8a3496e8bc5d0e24fbe3cf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:06.562697 22240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb942ebaa8a3496e8bc5d0e24fbe3cf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:06.569064 22469 tablet_bootstrap.cc:492] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5: Bootstrap starting.
I20260929 10:28:06.570994 22471 tablet_bootstrap.cc:492] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P bb662e153eb5403ea049e77809020c45: Bootstrap starting.
I20260929 10:28:06.571540 22470 tablet_bootstrap.cc:492] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P fb040c004af747e9806f728b7eeb3022: Bootstrap starting.
I20260929 10:28:06.572067 22469 tablet_bootstrap.cc:654] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:06.572836 22469 log.cc:824] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:06.573228 22471 tablet_bootstrap.cc:654] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P bb662e153eb5403ea049e77809020c45: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:06.573575 22470 tablet_bootstrap.cc:654] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P fb040c004af747e9806f728b7eeb3022: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:06.574112 22471 log.cc:824] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P bb662e153eb5403ea049e77809020c45: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:06.574339 22470 log.cc:824] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P fb040c004af747e9806f728b7eeb3022: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:06.574934 22469 tablet_bootstrap.cc:492] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5: No bootstrap required, opened a new log
I20260929 10:28:06.575155 22469 ts_tablet_manager.cc:1397] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 10:28:06.575872 22471 tablet_bootstrap.cc:492] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P bb662e153eb5403ea049e77809020c45: No bootstrap required, opened a new log
I20260929 10:28:06.576069 22471 ts_tablet_manager.cc:1397] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P bb662e153eb5403ea049e77809020c45: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 10:28:06.577287 22470 tablet_bootstrap.cc:492] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P fb040c004af747e9806f728b7eeb3022: No bootstrap required, opened a new log
I20260929 10:28:06.577471 22470 ts_tablet_manager.cc:1397] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P fb040c004af747e9806f728b7eeb3022: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260929 10:28:06.579252 22469 raft_consensus.cc:348] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } } peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } }
I20260929 10:28:06.579519 22469 raft_consensus.cc:374] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:06.579571 22469 raft_consensus.cc:729] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e91d274147b4563940974d85c4b34d5, State: Initialized, Role: FOLLOWER
I20260929 10:28:06.580006 22469 consensus_queue.cc:260] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } } peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } }
I20260929 10:28:06.579963 22471 raft_consensus.cc:348] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P bb662e153eb5403ea049e77809020c45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } } peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } }
I20260929 10:28:06.580214 22471 raft_consensus.cc:374] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P bb662e153eb5403ea049e77809020c45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:06.580267 22471 raft_consensus.cc:729] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P bb662e153eb5403ea049e77809020c45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb662e153eb5403ea049e77809020c45, State: Initialized, Role: FOLLOWER
I20260929 10:28:06.580740 22166 heartbeater.cc:503] Master 127.20.92.62:34829 was elected leader, sending a full tablet report...
I20260929 10:28:06.580705 22471 consensus_queue.cc:260] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P bb662e153eb5403ea049e77809020c45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } } peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } }
I20260929 10:28:06.581032 22469 ts_tablet_manager.cc:1428] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 10:28:06.581405 22446 heartbeater.cc:503] Master 127.20.92.62:34829 was elected leader, sending a full tablet report...
I20260929 10:28:06.581632 22471 ts_tablet_manager.cc:1428] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P bb662e153eb5403ea049e77809020c45: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 10:28:06.582077 22470 raft_consensus.cc:348] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P fb040c004af747e9806f728b7eeb3022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } } peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } }
I20260929 10:28:06.582345 22470 raft_consensus.cc:374] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P fb040c004af747e9806f728b7eeb3022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:06.582402 22470 raft_consensus.cc:729] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P fb040c004af747e9806f728b7eeb3022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb040c004af747e9806f728b7eeb3022, State: Initialized, Role: FOLLOWER
I20260929 10:28:06.582823 22470 consensus_queue.cc:260] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P fb040c004af747e9806f728b7eeb3022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } } peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } }
I20260929 10:28:06.583781 22306 heartbeater.cc:503] Master 127.20.92.62:34829 was elected leader, sending a full tablet report...
I20260929 10:28:06.584208 22470 ts_tablet_manager.cc:1428] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P fb040c004af747e9806f728b7eeb3022: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260929 10:28:06.739681 22447 tablet.cc:2369] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P bb662e153eb5403ea049e77809020c45: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:28:06.739681 22307 tablet.cc:2369] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P fb040c004af747e9806f728b7eeb3022: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:28:06.741160 22167 tablet.cc:2369] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:28:06.834877 22475 raft_consensus.cc:482] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:28:06.835088 22475 raft_consensus.cc:504] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } } peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } }
I20260929 10:28:06.836391 22475 leader_election.cc:288] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb662e153eb5403ea049e77809020c45 (127.20.92.3:40075), fb040c004af747e9806f728b7eeb3022 (127.20.92.2:37381)
I20260929 10:28:06.840785 22468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:06.836811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:33147 (local address 127.20.92.2:37381)
0929 10:28:06.836954 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:06.836960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:06.837155 (+ 195us) server_negotiation.cc:407] Connection header received
0929 10:28:06.837375 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:06.837382 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:06.837436 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:06.837501 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:06.837841 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.838433 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.839761 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.840164 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.840282 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:06.840365 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:06.840559 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 10:28:06.840642 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 10:28:06.840785 22465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:06.836766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:43701 (local address 127.20.92.3:40075)
0929 10:28:06.836908 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:06.836915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:06.837272 (+ 357us) server_negotiation.cc:407] Connection header received
0929 10:28:06.837427 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:06.837429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:06.837471 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:06.837810 (+ 339us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:06.838015 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.838534 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.839772 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.840179 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.840312 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:06.840376 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:06.840604 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 10:28:06.840676 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 10:28:06.840818 22479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:06.836661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:37381 (local address 127.20.92.1:33147)
0929 10:28:06.837058 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:06.837069 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:06.837221 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:06.837548 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:06.837553 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:06.837573 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:06.837711 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:06.837719 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.838549 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.838554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:06.839609 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:06.839620 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.840304 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.840308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:06.840425 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:06.840449 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 10:28:06.840519 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 10:28:06.840618 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":74,"thread_start_us":118,"threads_started":1}
I20260929 10:28:06.840855 22478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:06.836664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:40075 (local address 127.20.92.1:43701)
0929 10:28:06.837181 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:06.837192 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:06.837315 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:06.837738 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:06.837761 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:06.837783 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:06.837924 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:06.837930 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.838655 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.838658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:06.839571 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:06.839581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.840340 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.840343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:06.840457 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:06.840475 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 10:28:06.840580 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 10:28:06.840652 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":34,"thread_start_us":182,"threads_started":1}
I20260929 10:28:06.841562 22400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb942ebaa8a3496e8bc5d0e24fbe3cf7" candidate_uuid: "7e91d274147b4563940974d85c4b34d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb662e153eb5403ea049e77809020c45" is_pre_election: true
I20260929 10:28:06.841554 22260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb942ebaa8a3496e8bc5d0e24fbe3cf7" candidate_uuid: "7e91d274147b4563940974d85c4b34d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb040c004af747e9806f728b7eeb3022" is_pre_election: true
I20260929 10:28:06.841929 22400 raft_consensus.cc:2454] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P bb662e153eb5403ea049e77809020c45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e91d274147b4563940974d85c4b34d5 in term 0.
I20260929 10:28:06.841933 22260 raft_consensus.cc:2454] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P fb040c004af747e9806f728b7eeb3022 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e91d274147b4563940974d85c4b34d5 in term 0.
I20260929 10:28:06.842432 22053 leader_election.cc:302] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e91d274147b4563940974d85c4b34d5, bb662e153eb5403ea049e77809020c45; no voters:
I20260929 10:28:06.842744 22475 raft_consensus.cc:2781] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:28:06.842841 22475 raft_consensus.cc:482] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:28:06.842885 22475 raft_consensus.cc:3037] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:06.843990 22475 raft_consensus.cc:504] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } } peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } }
I20260929 10:28:06.844378 22475 leader_election.cc:288] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5 [CANDIDATE]: Term 1 election: Requested vote from peers bb662e153eb5403ea049e77809020c45 (127.20.92.3:40075), fb040c004af747e9806f728b7eeb3022 (127.20.92.2:37381)
I20260929 10:28:06.844774 22400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb942ebaa8a3496e8bc5d0e24fbe3cf7" candidate_uuid: "7e91d274147b4563940974d85c4b34d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb662e153eb5403ea049e77809020c45"
I20260929 10:28:06.844841 22260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb942ebaa8a3496e8bc5d0e24fbe3cf7" candidate_uuid: "7e91d274147b4563940974d85c4b34d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb040c004af747e9806f728b7eeb3022"
I20260929 10:28:06.844939 22400 raft_consensus.cc:3037] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P bb662e153eb5403ea049e77809020c45 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:06.844981 22260 raft_consensus.cc:3037] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P fb040c004af747e9806f728b7eeb3022 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:06.846196 22260 raft_consensus.cc:2454] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P fb040c004af747e9806f728b7eeb3022 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e91d274147b4563940974d85c4b34d5 in term 1.
I20260929 10:28:06.846642 22052 leader_election.cc:302] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e91d274147b4563940974d85c4b34d5, fb040c004af747e9806f728b7eeb3022; no voters:
I20260929 10:28:06.846783 22400 raft_consensus.cc:2454] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P bb662e153eb5403ea049e77809020c45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e91d274147b4563940974d85c4b34d5 in term 1.
I20260929 10:28:06.846866 22475 raft_consensus.cc:2781] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:06.847124 22475 raft_consensus.cc:686] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5 [term 1 LEADER]: Becoming Leader. State: Replica: 7e91d274147b4563940974d85c4b34d5, State: Running, Role: LEADER
I20260929 10:28:06.847575 22475 consensus_queue.cc:237] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } } peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } }
I20260929 10:28:06.851410 21961 catalog_manager.cc:5495] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e91d274147b4563940974d85c4b34d5 (127.20.92.1). New cstate: current_term: 1 leader_uuid: "7e91d274147b4563940974d85c4b34d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } health_report { overall_health: UNKNOWN } } }
I20260929 10:28:06.935446 22466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:06.931021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58616 (local address 127.20.92.1:36413)
0929 10:28:06.931150 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:06.931154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:06.931168 (+ 14us) server_negotiation.cc:407] Connection header received
0929 10:28:06.931358 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:06.931361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:06.931419 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:06.931503 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:28:06.931823 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.932334 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.933442 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:06.933684 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:06.933956 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:06.934387 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:28:06.934926 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:28:06.935034 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:06.935257 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 10:28:06.935323 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 10:28:06.941926 22260 raft_consensus.cc:1261] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P fb040c004af747e9806f728b7eeb3022 [term 1 FOLLOWER]: Refusing update from remote peer 7e91d274147b4563940974d85c4b34d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:06.941936 22400 raft_consensus.cc:1261] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P bb662e153eb5403ea049e77809020c45 [term 1 FOLLOWER]: Refusing update from remote peer 7e91d274147b4563940974d85c4b34d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:06.942646 22475 consensus_queue.cc:1035] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:28:06.942899 22480 consensus_queue.cc:1035] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:28:06.954027 22484 mvcc.cc:205] Tried to move back new op lower bound from 7334615805706354688 to 7334615805331292160. Current Snapshot: MvccSnapshot[applied={T|T < 7334615805706354688}]
I20260929 10:28:06.954274 22485 mvcc.cc:205] Tried to move back new op lower bound from 7334615805706354688 to 7334615805331292160. Current Snapshot: MvccSnapshot[applied={T|T < 7334615805706354688}]
I20260929 10:28:06.965664 21961 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:28:06.965979 21961 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:28:06.968895 21961 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70ba2352c99b489d95f2c7ae38dca8b2: Sending DeleteTablet for 3 replicas of tablet eb942ebaa8a3496e8bc5d0e24fbe3cf7
I20260929 10:28:06.969900 22380 tablet_service.cc:1514] Processing DeleteTablet for tablet eb942ebaa8a3496e8bc5d0e24fbe3cf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:06 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:52982
I20260929 10:28:06.969897 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet eb942ebaa8a3496e8bc5d0e24fbe3cf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:06 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:58608
I20260929 10:28:06.970067 22240 tablet_service.cc:1514] Processing DeleteTablet for tablet eb942ebaa8a3496e8bc5d0e24fbe3cf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:06 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:49806
I20260929 10:28:06.970608 22492 tablet_replica.cc:330] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P bb662e153eb5403ea049e77809020c45: stopping tablet replica
I20260929 10:28:06.970752 22494 tablet_replica.cc:330] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P fb040c004af747e9806f728b7eeb3022: stopping tablet replica
I20260929 10:28:06.971095 22494 raft_consensus.cc:2229] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P fb040c004af747e9806f728b7eeb3022 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:28:06.971096 22492 raft_consensus.cc:2229] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P bb662e153eb5403ea049e77809020c45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:28:06.971498 22492 raft_consensus.cc:2258] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P bb662e153eb5403ea049e77809020c45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:06.971565 22493 tablet_replica.cc:330] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5: stopping tablet replica
I20260929 10:28:06.971588 21961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 10:28:06.971894 22493 raft_consensus.cc:2229] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5 [term 1 LEADER]: Raft consensus shutting down.
W20260929 10:28:06.972078 21961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:28:06.972368 22493 raft_consensus.cc:2258] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:06.972692 22494 raft_consensus.cc:2258] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P fb040c004af747e9806f728b7eeb3022 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:06.974009 22492 ts_tablet_manager.cc:1905] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P bb662e153eb5403ea049e77809020c45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:06.974664 22493 ts_tablet_manager.cc:1905] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:06.976948 22492 ts_tablet_manager.cc:1918] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P bb662e153eb5403ea049e77809020c45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:28:06.977128 22492 log.cc:1196] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P bb662e153eb5403ea049e77809020c45: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-2/wal/wals/eb942ebaa8a3496e8bc5d0e24fbe3cf7
I20260929 10:28:06.977252 22493 ts_tablet_manager.cc:1918] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:28:06.977350 22493 log.cc:1196] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-0/wal/wals/eb942ebaa8a3496e8bc5d0e24fbe3cf7
I20260929 10:28:06.977662 22493 ts_tablet_manager.cc:1939] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P 7e91d274147b4563940974d85c4b34d5: Deleting consensus metadata
I20260929 10:28:06.977667 22492 ts_tablet_manager.cc:1939] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P bb662e153eb5403ea049e77809020c45: Deleting consensus metadata
I20260929 10:28:06.978506 22380 tablet_service.cc:1467] Processing CreateTablet for tablet bad5e7af94d4427dae709fdcc12abb67 (DEFAULT_TABLE table=test-table [id=1927791839614b21a0a98296cbcb366b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:06.978507 22240 tablet_service.cc:1467] Processing CreateTablet for tablet bad5e7af94d4427dae709fdcc12abb67 (DEFAULT_TABLE table=test-table [id=1927791839614b21a0a98296cbcb366b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:06.978777 22240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bad5e7af94d4427dae709fdcc12abb67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:06.978868 22380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bad5e7af94d4427dae709fdcc12abb67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:06.979133 21950 catalog_manager.cc:4841] TS 7e91d274147b4563940974d85c4b34d5 (127.20.92.1:36413): tablet eb942ebaa8a3496e8bc5d0e24fbe3cf7 (table test-table [id=0c7a5f85eae548cdb8b5791908648832]) successfully deleted
I20260929 10:28:06.979769 22100 tablet_service.cc:1467] Processing CreateTablet for tablet bad5e7af94d4427dae709fdcc12abb67 (DEFAULT_TABLE table=test-table [id=1927791839614b21a0a98296cbcb366b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:06.980036 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bad5e7af94d4427dae709fdcc12abb67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:06.980618 22494 ts_tablet_manager.cc:1905] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P fb040c004af747e9806f728b7eeb3022: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:06.982266 22470 tablet_bootstrap.cc:492] T bad5e7af94d4427dae709fdcc12abb67 P fb040c004af747e9806f728b7eeb3022: Bootstrap starting.
I20260929 10:28:06.983219 22494 ts_tablet_manager.cc:1918] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P fb040c004af747e9806f728b7eeb3022: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:28:06.983319 22494 log.cc:1196] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P fb040c004af747e9806f728b7eeb3022: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-1/wal/wals/eb942ebaa8a3496e8bc5d0e24fbe3cf7
I20260929 10:28:06.983448 22470 tablet_bootstrap.cc:654] T bad5e7af94d4427dae709fdcc12abb67 P fb040c004af747e9806f728b7eeb3022: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:06.983621 22494 ts_tablet_manager.cc:1939] T eb942ebaa8a3496e8bc5d0e24fbe3cf7 P fb040c004af747e9806f728b7eeb3022: Deleting consensus metadata
I20260929 10:28:06.984103 22469 tablet_bootstrap.cc:492] T bad5e7af94d4427dae709fdcc12abb67 P 7e91d274147b4563940974d85c4b34d5: Bootstrap starting.
I20260929 10:28:06.985112 22471 tablet_bootstrap.cc:492] T bad5e7af94d4427dae709fdcc12abb67 P bb662e153eb5403ea049e77809020c45: Bootstrap starting.
I20260929 10:28:06.985258 22469 tablet_bootstrap.cc:654] T bad5e7af94d4427dae709fdcc12abb67 P 7e91d274147b4563940974d85c4b34d5: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:06.986136 22471 tablet_bootstrap.cc:654] T bad5e7af94d4427dae709fdcc12abb67 P bb662e153eb5403ea049e77809020c45: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:06.987094 21949 catalog_manager.cc:4841] TS bb662e153eb5403ea049e77809020c45 (127.20.92.3:40075): tablet eb942ebaa8a3496e8bc5d0e24fbe3cf7 (table test-table [id=0c7a5f85eae548cdb8b5791908648832]) successfully deleted
I20260929 10:28:06.987536 21948 catalog_manager.cc:4841] TS fb040c004af747e9806f728b7eeb3022 (127.20.92.2:37381): tablet eb942ebaa8a3496e8bc5d0e24fbe3cf7 (table test-table [id=0c7a5f85eae548cdb8b5791908648832]) successfully deleted
I20260929 10:28:06.988276 22471 tablet_bootstrap.cc:492] T bad5e7af94d4427dae709fdcc12abb67 P bb662e153eb5403ea049e77809020c45: No bootstrap required, opened a new log
I20260929 10:28:06.988278 22470 tablet_bootstrap.cc:492] T bad5e7af94d4427dae709fdcc12abb67 P fb040c004af747e9806f728b7eeb3022: No bootstrap required, opened a new log
I20260929 10:28:06.988377 22470 ts_tablet_manager.cc:1397] T bad5e7af94d4427dae709fdcc12abb67 P fb040c004af747e9806f728b7eeb3022: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260929 10:28:06.988372 22471 ts_tablet_manager.cc:1397] T bad5e7af94d4427dae709fdcc12abb67 P bb662e153eb5403ea049e77809020c45: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 10:28:06.988482 22469 tablet_bootstrap.cc:492] T bad5e7af94d4427dae709fdcc12abb67 P 7e91d274147b4563940974d85c4b34d5: No bootstrap required, opened a new log
I20260929 10:28:06.988565 22469 ts_tablet_manager.cc:1397] T bad5e7af94d4427dae709fdcc12abb67 P 7e91d274147b4563940974d85c4b34d5: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260929 10:28:06.988860 22470 raft_consensus.cc:348] T bad5e7af94d4427dae709fdcc12abb67 P fb040c004af747e9806f728b7eeb3022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } } peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } }
I20260929 10:28:06.988859 22471 raft_consensus.cc:348] T bad5e7af94d4427dae709fdcc12abb67 P bb662e153eb5403ea049e77809020c45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } } peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } }
I20260929 10:28:06.988976 22470 raft_consensus.cc:374] T bad5e7af94d4427dae709fdcc12abb67 P fb040c004af747e9806f728b7eeb3022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:06.988976 22471 raft_consensus.cc:374] T bad5e7af94d4427dae709fdcc12abb67 P bb662e153eb5403ea049e77809020c45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:06.989009 22471 raft_consensus.cc:729] T bad5e7af94d4427dae709fdcc12abb67 P bb662e153eb5403ea049e77809020c45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb662e153eb5403ea049e77809020c45, State: Initialized, Role: FOLLOWER
I20260929 10:28:06.989009 22470 raft_consensus.cc:729] T bad5e7af94d4427dae709fdcc12abb67 P fb040c004af747e9806f728b7eeb3022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb040c004af747e9806f728b7eeb3022, State: Initialized, Role: FOLLOWER
I20260929 10:28:06.989015 22469 raft_consensus.cc:348] T bad5e7af94d4427dae709fdcc12abb67 P 7e91d274147b4563940974d85c4b34d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } } peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } }
I20260929 10:28:06.989125 22469 raft_consensus.cc:374] T bad5e7af94d4427dae709fdcc12abb67 P 7e91d274147b4563940974d85c4b34d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:06.989156 22469 raft_consensus.cc:729] T bad5e7af94d4427dae709fdcc12abb67 P 7e91d274147b4563940974d85c4b34d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e91d274147b4563940974d85c4b34d5, State: Initialized, Role: FOLLOWER
I20260929 10:28:06.989140 22470 consensus_queue.cc:260] T bad5e7af94d4427dae709fdcc12abb67 P fb040c004af747e9806f728b7eeb3022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } } peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } }
I20260929 10:28:06.989152 22471 consensus_queue.cc:260] T bad5e7af94d4427dae709fdcc12abb67 P bb662e153eb5403ea049e77809020c45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } } peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } }
I20260929 10:28:06.989271 22469 consensus_queue.cc:260] T bad5e7af94d4427dae709fdcc12abb67 P 7e91d274147b4563940974d85c4b34d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } } peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } }
I20260929 10:28:06.989394 22471 ts_tablet_manager.cc:1428] T bad5e7af94d4427dae709fdcc12abb67 P bb662e153eb5403ea049e77809020c45: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 10:28:06.989403 22470 ts_tablet_manager.cc:1428] T bad5e7af94d4427dae709fdcc12abb67 P fb040c004af747e9806f728b7eeb3022: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:28:06.989516 22469 ts_tablet_manager.cc:1428] T bad5e7af94d4427dae709fdcc12abb67 P 7e91d274147b4563940974d85c4b34d5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 10:28:07.166883 22477 raft_consensus.cc:482] T bad5e7af94d4427dae709fdcc12abb67 P fb040c004af747e9806f728b7eeb3022 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:28:07.167039 22477 raft_consensus.cc:504] T bad5e7af94d4427dae709fdcc12abb67 P fb040c004af747e9806f728b7eeb3022 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } } peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } }
I20260929 10:28:07.168267 22477 leader_election.cc:288] T bad5e7af94d4427dae709fdcc12abb67 P fb040c004af747e9806f728b7eeb3022 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e91d274147b4563940974d85c4b34d5 (127.20.92.1:36413), bb662e153eb5403ea049e77809020c45 (127.20.92.3:40075)
I20260929 10:28:07.174065 22465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:07.168696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:50817 (local address 127.20.92.3:40075)
0929 10:28:07.168916 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:07.168922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:07.169115 (+ 193us) server_negotiation.cc:407] Connection header received
0929 10:28:07.169268 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:07.169271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:07.169313 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:07.169397 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:07.169890 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:07.170660 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:07.172543 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:07.173232 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:07.173335 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:07.173408 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:07.173853 (+ 445us) server_negotiation.cc:299] Negotiation successful
0929 10:28:07.173937 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260929 10:28:07.174065 22466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:07.168691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:35295 (local address 127.20.92.1:36413)
0929 10:28:07.168919 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:07.168924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:07.169115 (+ 191us) server_negotiation.cc:407] Connection header received
0929 10:28:07.169252 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:07.169255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:07.169313 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:07.169418 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:07.169872 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:07.170661 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:07.172543 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:07.173189 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:07.173333 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:07.173409 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:07.173853 (+ 444us) server_negotiation.cc:299] Negotiation successful
0929 10:28:07.173937 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260929 10:28:07.174090 22498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:07.168571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:40075 (local address 127.20.92.2:50817)
0929 10:28:07.168953 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:07.168970 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:07.169128 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:07.169447 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:07.169450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:07.169479 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:07.169713 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:07.169719 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:07.170815 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:07.170824 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:07.172359 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:07.172375 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:07.173434 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:07.173439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:07.173616 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:07.173669 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 10:28:07.173783 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 10:28:07.173894 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":185,"spinlock_wait_cycles":22400,"thread_start_us":72,"threads_started":1}
I20260929 10:28:07.174098 22499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:07.168571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:36413 (local address 127.20.92.2:35295)
0929 10:28:07.169003 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:07.169016 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:07.169128 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:07.169447 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:07.169450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:07.169479 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:07.169706 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:07.169714 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:07.170815 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:07.170819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:07.172376 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:07.172383 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:07.173434 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:07.173439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:07.173619 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:07.173659 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 10:28:07.173783 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 10:28:07.173915 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":304,"spinlock_wait_cycles":21120,"thread_start_us":143,"threads_started":1}
I20260929 10:28:07.174690 22400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bad5e7af94d4427dae709fdcc12abb67" candidate_uuid: "fb040c004af747e9806f728b7eeb3022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb662e153eb5403ea049e77809020c45" is_pre_election: true
I20260929 10:28:07.174865 22400 raft_consensus.cc:2454] T bad5e7af94d4427dae709fdcc12abb67 P bb662e153eb5403ea049e77809020c45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb040c004af747e9806f728b7eeb3022 in term 0.
I20260929 10:28:07.174960 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bad5e7af94d4427dae709fdcc12abb67" candidate_uuid: "fb040c004af747e9806f728b7eeb3022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e91d274147b4563940974d85c4b34d5" is_pre_election: true
I20260929 10:28:07.175227 22120 raft_consensus.cc:2454] T bad5e7af94d4427dae709fdcc12abb67 P 7e91d274147b4563940974d85c4b34d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb040c004af747e9806f728b7eeb3022 in term 0.
I20260929 10:28:07.175292 22193 leader_election.cc:302] T bad5e7af94d4427dae709fdcc12abb67 P fb040c004af747e9806f728b7eeb3022 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb662e153eb5403ea049e77809020c45, fb040c004af747e9806f728b7eeb3022; no voters:
I20260929 10:28:07.175624 22477 raft_consensus.cc:2781] T bad5e7af94d4427dae709fdcc12abb67 P fb040c004af747e9806f728b7eeb3022 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:28:07.175752 22477 raft_consensus.cc:482] T bad5e7af94d4427dae709fdcc12abb67 P fb040c004af747e9806f728b7eeb3022 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:28:07.175827 22477 raft_consensus.cc:3037] T bad5e7af94d4427dae709fdcc12abb67 P fb040c004af747e9806f728b7eeb3022 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:07.176826 22477 raft_consensus.cc:504] T bad5e7af94d4427dae709fdcc12abb67 P fb040c004af747e9806f728b7eeb3022 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } } peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } }
I20260929 10:28:07.177225 22477 leader_election.cc:288] T bad5e7af94d4427dae709fdcc12abb67 P fb040c004af747e9806f728b7eeb3022 [CANDIDATE]: Term 1 election: Requested vote from peers 7e91d274147b4563940974d85c4b34d5 (127.20.92.1:36413), bb662e153eb5403ea049e77809020c45 (127.20.92.3:40075)
I20260929 10:28:07.177634 22400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bad5e7af94d4427dae709fdcc12abb67" candidate_uuid: "fb040c004af747e9806f728b7eeb3022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb662e153eb5403ea049e77809020c45"
I20260929 10:28:07.177793 22400 raft_consensus.cc:3037] T bad5e7af94d4427dae709fdcc12abb67 P bb662e153eb5403ea049e77809020c45 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:07.177680 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bad5e7af94d4427dae709fdcc12abb67" candidate_uuid: "fb040c004af747e9806f728b7eeb3022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e91d274147b4563940974d85c4b34d5"
I20260929 10:28:07.177953 22120 raft_consensus.cc:3037] T bad5e7af94d4427dae709fdcc12abb67 P 7e91d274147b4563940974d85c4b34d5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:07.178799 22400 raft_consensus.cc:2454] T bad5e7af94d4427dae709fdcc12abb67 P bb662e153eb5403ea049e77809020c45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb040c004af747e9806f728b7eeb3022 in term 1.
I20260929 10:28:07.179201 22120 raft_consensus.cc:2454] T bad5e7af94d4427dae709fdcc12abb67 P 7e91d274147b4563940974d85c4b34d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb040c004af747e9806f728b7eeb3022 in term 1.
I20260929 10:28:07.179180 22193 leader_election.cc:302] T bad5e7af94d4427dae709fdcc12abb67 P fb040c004af747e9806f728b7eeb3022 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb662e153eb5403ea049e77809020c45, fb040c004af747e9806f728b7eeb3022; no voters:
I20260929 10:28:07.179577 22477 raft_consensus.cc:2781] T bad5e7af94d4427dae709fdcc12abb67 P fb040c004af747e9806f728b7eeb3022 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:07.180042 22477 raft_consensus.cc:686] T bad5e7af94d4427dae709fdcc12abb67 P fb040c004af747e9806f728b7eeb3022 [term 1 LEADER]: Becoming Leader. State: Replica: fb040c004af747e9806f728b7eeb3022, State: Running, Role: LEADER
I20260929 10:28:07.180475 22477 consensus_queue.cc:237] T bad5e7af94d4427dae709fdcc12abb67 P fb040c004af747e9806f728b7eeb3022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } } peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } }
I20260929 10:28:07.183301 21963 catalog_manager.cc:5495] T bad5e7af94d4427dae709fdcc12abb67 P fb040c004af747e9806f728b7eeb3022 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb040c004af747e9806f728b7eeb3022 (127.20.92.2). New cstate: current_term: 1 leader_uuid: "fb040c004af747e9806f728b7eeb3022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } health_report { overall_health: UNKNOWN } } }
Sep 29 10:28:07 dist-test-slave-9cj5 krb5kdc[21918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677685, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:07.262014 22507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:07.251667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:34829 (local address 127.0.0.1:36926)
0929 10:28:07.251964 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:07.251982 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:07.252134 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:07.252464 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:07.252466 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:07.252876 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:07.252981 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:07.252988 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:07.253952 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:07.253955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:07.254502 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:07.254509 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:07.255085 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:07.255088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:07.255153 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:07.255817 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:07.255830 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:07.257862 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:07.259936 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:07.259948 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:07.259957 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:07.260262 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:07.260658 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:07.260661 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:07.260662 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:07.260832 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:07.261362 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:07.261368 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:07.261484 (+ 116us) client_negotiation.cc:769] Sending connection context
0929 10:28:07.261592 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 10:28:07.261851 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":123,"threads_started":1}
I20260929 10:28:07.262200 22508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:07.251720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36926 (local address 127.20.92.62:34829)
0929 10:28:07.251995 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:07.252004 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:07.252111 (+ 107us) server_negotiation.cc:407] Connection header received
0929 10:28:07.252237 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:07.252245 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:07.252290 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:07.252443 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:07.253126 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:07.253848 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:07.254642 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:07.254933 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:07.255027 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:07.258041 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:07.258044 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:07.258046 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:07.258106 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:07.259775 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:07.260421 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:07.260424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:07.260426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:07.260531 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:07.260975 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:07.260978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:07.260980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:07.261200 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:07.261319 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:07.261670 (+ 351us) server_negotiation.cc:299] Negotiation successful
0929 10:28:07.262001 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":187,"server-negotiator.queue_time_us":207,"thread_start_us":111,"threads_started":1}
I20260929 10:28:07.266577 21963 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:36926:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 10:28:07.267342 21963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:28:07.272722 22100 tablet_service.cc:1467] Processing CreateTablet for tablet 52b94041f25d404b8ef4d4823ded9485 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3bc908766eb246b2a1f0285538e39179]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 10:28:07.272974 22240 tablet_service.cc:1467] Processing CreateTablet for tablet 52b94041f25d404b8ef4d4823ded9485 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3bc908766eb246b2a1f0285538e39179]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 10:28:07.273133 22380 tablet_service.cc:1467] Processing CreateTablet for tablet 52b94041f25d404b8ef4d4823ded9485 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3bc908766eb246b2a1f0285538e39179]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 10:28:07.273270 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52b94041f25d404b8ef4d4823ded9485. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:07.273432 22240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52b94041f25d404b8ef4d4823ded9485. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:07.273517 22380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52b94041f25d404b8ef4d4823ded9485. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:07.276067 22470 tablet_bootstrap.cc:492] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022: Bootstrap starting.
I20260929 10:28:07.276173 22469 tablet_bootstrap.cc:492] T 52b94041f25d404b8ef4d4823ded9485 P 7e91d274147b4563940974d85c4b34d5: Bootstrap starting.
I20260929 10:28:07.277084 22469 tablet_bootstrap.cc:654] T 52b94041f25d404b8ef4d4823ded9485 P 7e91d274147b4563940974d85c4b34d5: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:07.277086 22470 tablet_bootstrap.cc:654] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:07.277926 22471 tablet_bootstrap.cc:492] T 52b94041f25d404b8ef4d4823ded9485 P bb662e153eb5403ea049e77809020c45: Bootstrap starting.
I20260929 10:28:07.278277 22469 tablet_bootstrap.cc:492] T 52b94041f25d404b8ef4d4823ded9485 P 7e91d274147b4563940974d85c4b34d5: No bootstrap required, opened a new log
I20260929 10:28:07.278374 22470 tablet_bootstrap.cc:492] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022: No bootstrap required, opened a new log
I20260929 10:28:07.278376 22469 ts_tablet_manager.cc:1397] T 52b94041f25d404b8ef4d4823ded9485 P 7e91d274147b4563940974d85c4b34d5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260929 10:28:07.278440 22470 ts_tablet_manager.cc:1397] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:28:07.278785 22469 raft_consensus.cc:348] T 52b94041f25d404b8ef4d4823ded9485 P 7e91d274147b4563940974d85c4b34d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } } peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } }
I20260929 10:28:07.278882 22471 tablet_bootstrap.cc:654] T 52b94041f25d404b8ef4d4823ded9485 P bb662e153eb5403ea049e77809020c45: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:07.278905 22469 raft_consensus.cc:374] T 52b94041f25d404b8ef4d4823ded9485 P 7e91d274147b4563940974d85c4b34d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:07.278937 22469 raft_consensus.cc:729] T 52b94041f25d404b8ef4d4823ded9485 P 7e91d274147b4563940974d85c4b34d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e91d274147b4563940974d85c4b34d5, State: Initialized, Role: FOLLOWER
I20260929 10:28:07.278862 22470 raft_consensus.cc:348] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } } peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } }
I20260929 10:28:07.279002 22470 raft_consensus.cc:374] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:07.279040 22470 raft_consensus.cc:729] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb040c004af747e9806f728b7eeb3022, State: Initialized, Role: FOLLOWER
I20260929 10:28:07.279073 22469 consensus_queue.cc:260] T 52b94041f25d404b8ef4d4823ded9485 P 7e91d274147b4563940974d85c4b34d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } } peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } }
I20260929 10:28:07.279178 22470 consensus_queue.cc:260] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } } peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } }
I20260929 10:28:07.279336 22469 ts_tablet_manager.cc:1428] T 52b94041f25d404b8ef4d4823ded9485 P 7e91d274147b4563940974d85c4b34d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:28:07.279453 22477 tablet_replica.cc:439] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } } peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } } }
I20260929 10:28:07.279532 22480 tablet_replica.cc:439] T 52b94041f25d404b8ef4d4823ded9485 P 7e91d274147b4563940974d85c4b34d5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } } peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } } }
I20260929 10:28:07.279601 22477 tablet_replica.cc:442] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:28:07.279645 22480 tablet_replica.cc:442] T 52b94041f25d404b8ef4d4823ded9485 P 7e91d274147b4563940974d85c4b34d5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:28:07.280318 22470 ts_tablet_manager.cc:1428] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:28:07.280674 22471 tablet_bootstrap.cc:492] T 52b94041f25d404b8ef4d4823ded9485 P bb662e153eb5403ea049e77809020c45: No bootstrap required, opened a new log
I20260929 10:28:07.280790 22471 ts_tablet_manager.cc:1397] T 52b94041f25d404b8ef4d4823ded9485 P bb662e153eb5403ea049e77809020c45: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260929 10:28:07.281297 22471 raft_consensus.cc:348] T 52b94041f25d404b8ef4d4823ded9485 P bb662e153eb5403ea049e77809020c45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } } peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } }
I20260929 10:28:07.281448 22471 raft_consensus.cc:374] T 52b94041f25d404b8ef4d4823ded9485 P bb662e153eb5403ea049e77809020c45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:07.281489 22471 raft_consensus.cc:729] T 52b94041f25d404b8ef4d4823ded9485 P bb662e153eb5403ea049e77809020c45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb662e153eb5403ea049e77809020c45, State: Initialized, Role: FOLLOWER
I20260929 10:28:07.281612 22471 consensus_queue.cc:260] T 52b94041f25d404b8ef4d4823ded9485 P bb662e153eb5403ea049e77809020c45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } } peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } }
I20260929 10:28:07.281867 22476 tablet_replica.cc:439] T 52b94041f25d404b8ef4d4823ded9485 P bb662e153eb5403ea049e77809020c45: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } } peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } } }
I20260929 10:28:07.282011 22476 tablet_replica.cc:442] T 52b94041f25d404b8ef4d4823ded9485 P bb662e153eb5403ea049e77809020c45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:28:07.282223 22471 ts_tablet_manager.cc:1428] T 52b94041f25d404b8ef4d4823ded9485 P bb662e153eb5403ea049e77809020c45: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 10:28:07.324918 22477 raft_consensus.cc:482] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:28:07.325062 22477 raft_consensus.cc:504] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } } peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } }
I20260929 10:28:07.325547 22477 leader_election.cc:288] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e91d274147b4563940974d85c4b34d5 (127.20.92.1:36413), bb662e153eb5403ea049e77809020c45 (127.20.92.3:40075)
I20260929 10:28:07.326161 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52b94041f25d404b8ef4d4823ded9485" candidate_uuid: "fb040c004af747e9806f728b7eeb3022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e91d274147b4563940974d85c4b34d5" is_pre_election: true
I20260929 10:28:07.326167 22400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52b94041f25d404b8ef4d4823ded9485" candidate_uuid: "fb040c004af747e9806f728b7eeb3022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb662e153eb5403ea049e77809020c45" is_pre_election: true
I20260929 10:28:07.326342 22400 raft_consensus.cc:2454] T 52b94041f25d404b8ef4d4823ded9485 P bb662e153eb5403ea049e77809020c45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb040c004af747e9806f728b7eeb3022 in term 0.
I20260929 10:28:07.326335 22120 raft_consensus.cc:2454] T 52b94041f25d404b8ef4d4823ded9485 P 7e91d274147b4563940974d85c4b34d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb040c004af747e9806f728b7eeb3022 in term 0.
I20260929 10:28:07.326713 22194 leader_election.cc:302] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e91d274147b4563940974d85c4b34d5, fb040c004af747e9806f728b7eeb3022; no voters:
I20260929 10:28:07.326915 22477 raft_consensus.cc:2781] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:28:07.326980 22477 raft_consensus.cc:482] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:28:07.327035 22477 raft_consensus.cc:3037] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:07.328094 22477 raft_consensus.cc:504] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } } peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } }
I20260929 10:28:07.328487 22477 leader_election.cc:288] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022 [CANDIDATE]: Term 1 election: Requested vote from peers 7e91d274147b4563940974d85c4b34d5 (127.20.92.1:36413), bb662e153eb5403ea049e77809020c45 (127.20.92.3:40075)
I20260929 10:28:07.328855 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52b94041f25d404b8ef4d4823ded9485" candidate_uuid: "fb040c004af747e9806f728b7eeb3022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e91d274147b4563940974d85c4b34d5"
I20260929 10:28:07.328928 22400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52b94041f25d404b8ef4d4823ded9485" candidate_uuid: "fb040c004af747e9806f728b7eeb3022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb662e153eb5403ea049e77809020c45"
I20260929 10:28:07.329003 22120 raft_consensus.cc:3037] T 52b94041f25d404b8ef4d4823ded9485 P 7e91d274147b4563940974d85c4b34d5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:07.329114 22400 raft_consensus.cc:3037] T 52b94041f25d404b8ef4d4823ded9485 P bb662e153eb5403ea049e77809020c45 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:07.330070 22120 raft_consensus.cc:2454] T 52b94041f25d404b8ef4d4823ded9485 P 7e91d274147b4563940974d85c4b34d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb040c004af747e9806f728b7eeb3022 in term 1.
I20260929 10:28:07.330155 22400 raft_consensus.cc:2454] T 52b94041f25d404b8ef4d4823ded9485 P bb662e153eb5403ea049e77809020c45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb040c004af747e9806f728b7eeb3022 in term 1.
I20260929 10:28:07.330401 22194 leader_election.cc:302] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e91d274147b4563940974d85c4b34d5, fb040c004af747e9806f728b7eeb3022; no voters:
I20260929 10:28:07.330595 22477 raft_consensus.cc:2781] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:07.330757 22477 raft_consensus.cc:686] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022 [term 1 LEADER]: Becoming Leader. State: Replica: fb040c004af747e9806f728b7eeb3022, State: Running, Role: LEADER
I20260929 10:28:07.330899 22477 consensus_queue.cc:237] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } } peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } }
I20260929 10:28:07.331585 22500 tablet_replica.cc:439] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022: TxnStatusTablet state changed. Reason: New leader fb040c004af747e9806f728b7eeb3022. Latest consensus state: current_term: 1 leader_uuid: "fb040c004af747e9806f728b7eeb3022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } } peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } } }
I20260929 10:28:07.331732 22500 tablet_replica.cc:442] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022: This TxnStatusTablet replica's current role is: LEADER
I20260929 10:28:07.332031 22513 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 10:28:07.332216 22513 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 10:28:07.332790 21962 catalog_manager.cc:5495] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb040c004af747e9806f728b7eeb3022 (127.20.92.2). New cstate: current_term: 1 leader_uuid: "fb040c004af747e9806f728b7eeb3022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } health_report { overall_health: UNKNOWN } } }
I20260929 10:28:07.350950 22516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:07.346678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49820 (local address 127.20.92.2:37381)
0929 10:28:07.346964 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:07.346968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:07.346981 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:28:07.347047 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:07.347049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:07.347142 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:07.347281 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:28:07.347576 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:07.348143 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:07.349224 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:07.349538 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:07.349645 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:07.350043 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:28:07.350513 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:28:07.350637 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:07.350707 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 10:28:07.350771 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":81,"threads_started":1}
I20260929 10:28:07.350934 22507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:07.346435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:37381 (local address 127.0.0.1:49820)
0929 10:28:07.346676 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:07.346691 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:07.346798 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:07.347310 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:07.347313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:07.347324 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:28:07.347452 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:07.347458 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:07.348284 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:07.348287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:07.349106 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:07.349116 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:07.349659 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:07.349662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:07.349795 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:07.349857 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:28:07.350611 (+ 754us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:28:07.350627 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 10:28:07.350699 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 10:28:07.350803 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260929 10:28:07.353979 22400 raft_consensus.cc:1261] T 52b94041f25d404b8ef4d4823ded9485 P bb662e153eb5403ea049e77809020c45 [term 1 FOLLOWER]: Refusing update from remote peer fb040c004af747e9806f728b7eeb3022: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:07.354277 22120 raft_consensus.cc:1261] T 52b94041f25d404b8ef4d4823ded9485 P 7e91d274147b4563940974d85c4b34d5 [term 1 FOLLOWER]: Refusing update from remote peer fb040c004af747e9806f728b7eeb3022: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:07.354522 22477 consensus_queue.cc:1035] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:28:07.354839 22500 consensus_queue.cc:1035] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:28:07.356009 22476 tablet_replica.cc:439] T 52b94041f25d404b8ef4d4823ded9485 P bb662e153eb5403ea049e77809020c45: TxnStatusTablet state changed. Reason: New leader fb040c004af747e9806f728b7eeb3022. Latest consensus state: current_term: 1 leader_uuid: "fb040c004af747e9806f728b7eeb3022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } } peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } } }
I20260929 10:28:07.356137 22476 tablet_replica.cc:442] T 52b94041f25d404b8ef4d4823ded9485 P bb662e153eb5403ea049e77809020c45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:28:07.356510 22480 tablet_replica.cc:439] T 52b94041f25d404b8ef4d4823ded9485 P 7e91d274147b4563940974d85c4b34d5: TxnStatusTablet state changed. Reason: New leader fb040c004af747e9806f728b7eeb3022. Latest consensus state: current_term: 1 leader_uuid: "fb040c004af747e9806f728b7eeb3022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } } peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } } }
I20260929 10:28:07.356645 22480 tablet_replica.cc:442] T 52b94041f25d404b8ef4d4823ded9485 P 7e91d274147b4563940974d85c4b34d5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:28:07.358783 22500 tablet_replica.cc:439] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb040c004af747e9806f728b7eeb3022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } } peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } } }
I20260929 10:28:07.358903 22500 tablet_replica.cc:442] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022: This TxnStatusTablet replica's current role is: LEADER
I20260929 10:28:07.359787 22477 tablet_replica.cc:439] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb040c004af747e9806f728b7eeb3022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } } peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } } }
I20260929 10:28:07.359894 22477 tablet_replica.cc:442] T 52b94041f25d404b8ef4d4823ded9485 P fb040c004af747e9806f728b7eeb3022: This TxnStatusTablet replica's current role is: LEADER
I20260929 10:28:07.360677 22483 mvcc.cc:205] Tried to move back new op lower bound from 7334615807396687872 to 7334615807308713984. Current Snapshot: MvccSnapshot[applied={T|T < 7334615807396687872}]
I20260929 10:28:07.360854 22480 tablet_replica.cc:439] T 52b94041f25d404b8ef4d4823ded9485 P 7e91d274147b4563940974d85c4b34d5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb040c004af747e9806f728b7eeb3022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } } peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } } }
I20260929 10:28:07.360946 22480 tablet_replica.cc:442] T 52b94041f25d404b8ef4d4823ded9485 P 7e91d274147b4563940974d85c4b34d5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:28:07.362303 22476 tablet_replica.cc:439] T 52b94041f25d404b8ef4d4823ded9485 P bb662e153eb5403ea049e77809020c45: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb040c004af747e9806f728b7eeb3022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 } } peers { permanent_uuid: "fb040c004af747e9806f728b7eeb3022" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 37381 } } peers { permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 } } }
I20260929 10:28:07.362448 22476 tablet_replica.cc:442] T 52b94041f25d404b8ef4d4823ded9485 P bb662e153eb5403ea049e77809020c45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:28:07.370504 22516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:07.366962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49836 (local address 127.20.92.2:37381)
0929 10:28:07.367079 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:07.367083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:07.367096 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:28:07.367243 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:07.367246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:07.367286 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:07.367354 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:28:07.367616 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:07.368116 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:07.369317 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:07.369648 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:07.369866 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:07.370043 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:28:07.370114 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:28:07.370176 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:07.370293 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 10:28:07.370340 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 10:28:07.382546 22516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:07.378305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:59681 (local address 127.20.92.2:37381)
0929 10:28:07.378433 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:07.378437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:07.378636 (+ 199us) server_negotiation.cc:407] Connection header received
0929 10:28:07.378722 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:07.378724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:07.378774 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:07.378843 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:28:07.379705 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:07.380219 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:07.381474 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:07.381683 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:07.381839 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:07.382086 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:28:07.382152 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:28:07.382207 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:07.382372 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 10:28:07.382419 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":56}
I20260929 10:28:07.382587 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:07.378161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:37381 (local address 127.20.92.2:59681)
0929 10:28:07.378510 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:07.378526 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:07.378644 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:07.378880 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:07.378883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:07.378890 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:28:07.379584 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:07.379593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:07.380335 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:07.380339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:07.381359 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:07.381370 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:07.381801 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:07.381805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:07.381898 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:07.381955 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:28:07.382233 (+ 278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:28:07.382239 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 10:28:07.382305 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 10:28:07.382393 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":23,"thread_start_us":122,"threads_started":1}
I20260929 10:28:07.390172 22400 raft_consensus.cc:1261] T bad5e7af94d4427dae709fdcc12abb67 P bb662e153eb5403ea049e77809020c45 [term 1 FOLLOWER]: Refusing update from remote peer fb040c004af747e9806f728b7eeb3022: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:07.390622 22120 raft_consensus.cc:1261] T bad5e7af94d4427dae709fdcc12abb67 P 7e91d274147b4563940974d85c4b34d5 [term 1 FOLLOWER]: Refusing update from remote peer fb040c004af747e9806f728b7eeb3022: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:07.390652 22522 consensus_queue.cc:1035] T bad5e7af94d4427dae709fdcc12abb67 P fb040c004af747e9806f728b7eeb3022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb662e153eb5403ea049e77809020c45" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:28:07.390957 22522 consensus_queue.cc:1035] T bad5e7af94d4427dae709fdcc12abb67 P fb040c004af747e9806f728b7eeb3022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e91d274147b4563940974d85c4b34d5" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 36413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:28:07.429911 21962 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:28:07.430096 21962 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:28:07.432343 21962 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70ba2352c99b489d95f2c7ae38dca8b2: Sending DeleteTablet for 3 replicas of tablet bad5e7af94d4427dae709fdcc12abb67
I20260929 10:28:07.433008 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet bad5e7af94d4427dae709fdcc12abb67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:07 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:58608
I20260929 10:28:07.433120 22380 tablet_service.cc:1514] Processing DeleteTablet for tablet bad5e7af94d4427dae709fdcc12abb67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:07 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:52982
I20260929 10:28:07.433243 22493 tablet_replica.cc:330] T bad5e7af94d4427dae709fdcc12abb67 P 7e91d274147b4563940974d85c4b34d5: stopping tablet replica
I20260929 10:28:07.433300 22492 tablet_replica.cc:330] T bad5e7af94d4427dae709fdcc12abb67 P bb662e153eb5403ea049e77809020c45: stopping tablet replica
I20260929 10:28:07.433385 22493 raft_consensus.cc:2229] T bad5e7af94d4427dae709fdcc12abb67 P 7e91d274147b4563940974d85c4b34d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:28:07.433423 22492 raft_consensus.cc:2229] T bad5e7af94d4427dae709fdcc12abb67 P bb662e153eb5403ea049e77809020c45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:28:07.433532 22492 raft_consensus.cc:2258] T bad5e7af94d4427dae709fdcc12abb67 P bb662e153eb5403ea049e77809020c45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:07.433530 22493 raft_consensus.cc:2258] T bad5e7af94d4427dae709fdcc12abb67 P 7e91d274147b4563940974d85c4b34d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:07.433928 22240 tablet_service.cc:1514] Processing DeleteTablet for tablet bad5e7af94d4427dae709fdcc12abb67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:07 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:49806
I20260929 10:28:07.434113 22494 tablet_replica.cc:330] T bad5e7af94d4427dae709fdcc12abb67 P fb040c004af747e9806f728b7eeb3022: stopping tablet replica
I20260929 10:28:07.434227 22494 raft_consensus.cc:2229] T bad5e7af94d4427dae709fdcc12abb67 P fb040c004af747e9806f728b7eeb3022 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:28:07.434463 22492 ts_tablet_manager.cc:1905] T bad5e7af94d4427dae709fdcc12abb67 P bb662e153eb5403ea049e77809020c45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:07.434530 22494 raft_consensus.cc:2258] T bad5e7af94d4427dae709fdcc12abb67 P fb040c004af747e9806f728b7eeb3022 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:07.435418 22494 ts_tablet_manager.cc:1905] T bad5e7af94d4427dae709fdcc12abb67 P fb040c004af747e9806f728b7eeb3022: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:07.435734 22493 ts_tablet_manager.cc:1905] T bad5e7af94d4427dae709fdcc12abb67 P 7e91d274147b4563940974d85c4b34d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:07.437443 22492 ts_tablet_manager.cc:1918] T bad5e7af94d4427dae709fdcc12abb67 P bb662e153eb5403ea049e77809020c45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:28:07.437510 22492 log.cc:1196] T bad5e7af94d4427dae709fdcc12abb67 P bb662e153eb5403ea049e77809020c45: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-2/wal/wals/bad5e7af94d4427dae709fdcc12abb67
I20260929 10:28:07.437786 22492 ts_tablet_manager.cc:1939] T bad5e7af94d4427dae709fdcc12abb67 P bb662e153eb5403ea049e77809020c45: Deleting consensus metadata
I20260929 10:28:07.438138 22494 ts_tablet_manager.cc:1918] T bad5e7af94d4427dae709fdcc12abb67 P fb040c004af747e9806f728b7eeb3022: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:28:07.438227 22494 log.cc:1196] T bad5e7af94d4427dae709fdcc12abb67 P fb040c004af747e9806f728b7eeb3022: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-1/wal/wals/bad5e7af94d4427dae709fdcc12abb67
I20260929 10:28:07.438490 22494 ts_tablet_manager.cc:1939] T bad5e7af94d4427dae709fdcc12abb67 P fb040c004af747e9806f728b7eeb3022: Deleting consensus metadata
I20260929 10:28:07.438599 22493 ts_tablet_manager.cc:1918] T bad5e7af94d4427dae709fdcc12abb67 P 7e91d274147b4563940974d85c4b34d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:28:07.438678 22493 log.cc:1196] T bad5e7af94d4427dae709fdcc12abb67 P 7e91d274147b4563940974d85c4b34d5: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677679287822-20848-0/minicluster-data/ts-0/wal/wals/bad5e7af94d4427dae709fdcc12abb67
I20260929 10:28:07.438922 21949 catalog_manager.cc:4841] TS bb662e153eb5403ea049e77809020c45 (127.20.92.3:40075): tablet bad5e7af94d4427dae709fdcc12abb67 (table test-table [id=1927791839614b21a0a98296cbcb366b]) successfully deleted
I20260929 10:28:07.439256 22493 ts_tablet_manager.cc:1939] T bad5e7af94d4427dae709fdcc12abb67 P 7e91d274147b4563940974d85c4b34d5: Deleting consensus metadata
I20260929 10:28:07.439838 21948 catalog_manager.cc:4841] TS fb040c004af747e9806f728b7eeb3022 (127.20.92.2:37381): tablet bad5e7af94d4427dae709fdcc12abb67 (table test-table [id=1927791839614b21a0a98296cbcb366b]) successfully deleted
I20260929 10:28:07.440100 21950 catalog_manager.cc:4841] TS 7e91d274147b4563940974d85c4b34d5 (127.20.92.1:36413): tablet bad5e7af94d4427dae709fdcc12abb67 (table test-table [id=1927791839614b21a0a98296cbcb366b]) successfully deleted
Sep 29 10:28:07 dist-test-slave-9cj5 krb5kdc[21918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677686, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.92.1@KRBTEST.COM
I20260929 10:28:07.452630 22466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:07.440346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58620 (local address 127.20.92.1:36413)
0929 10:28:07.440471 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:07.440476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:07.440728 (+ 252us) server_negotiation.cc:407] Connection header received
0929 10:28:07.440832 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:07.440836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:07.440888 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:07.440980 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:07.441608 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:07.442492 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:07.443441 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:07.443835 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:07.443947 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:07.447817 (+ 3870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:07.447836 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:07.447854 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:07.447918 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:07.450344 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:07.450938 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:07.450946 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:07.450948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:07.451057 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:07.451463 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:07.451466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:07.451468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:07.451807 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:07.452050 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:07.452319 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 10:28:07.452461 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260929 10:28:07.453230 22160 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58620
I20260929 10:28:07.464877 22508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:07.455275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36942 (local address 127.20.92.62:34829)
0929 10:28:07.455417 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:07.455429 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:07.455447 (+ 18us) server_negotiation.cc:407] Connection header received
0929 10:28:07.455593 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:07.455597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:07.455650 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:07.455749 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:07.456569 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:07.457515 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:07.458536 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:07.458973 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:07.459105 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:07.460816 (+ 1711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:07.460821 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:07.460825 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:07.460861 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:07.462947 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:07.463455 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:07.463459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:07.463461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:07.463566 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:07.463860 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:07.463864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:07.463866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:07.464129 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:07.464291 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:07.464524 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 10:28:07.464680 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 10:28:07.465366 21962 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36942
I20260929 10:28:07.466917 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 22035
I20260929 10:28:07.475418 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 22175
I20260929 10:28:07.482164 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 22315
I20260929 10:28:07.488171 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 21931
2026-09-29T10:28:07Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3241 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:28:07 dist-test-slave-9cj5 krb5kdc[22533](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:28:07 dist-test-slave-9cj5 krb5kdc[22533](info): set up 2 sockets
Sep 29 10:28:07 dist-test-slave-9cj5 krb5kdc[22533](info): commencing operation
krb5kdc: starting...
W20260929 10:28:08.598564 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:28:08 dist-test-slave-9cj5 krb5kdc[22533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:28:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:28:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
I20260929 10:28:08.730902 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:38219
--webserver_interface=127.20.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:40617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:38219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:08.846134 22546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:08.846433 22546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:08.846477 22546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:08.850979 22546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:28:08.851064 22546 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 10:28:08.851083 22546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:08.851102 22546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:08.851120 22546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:28:08.851136 22546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:28:08.857138 22546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:40617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:38219
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:38219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:08.858296 22546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:08.859287 22546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:08.868096 22552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:08.868136 22551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:08.868103 22554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:08.868180 22546 server_base.cc:1034] running on GCE node
I20260929 10:28:08.868992 22546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:08.870242 22546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:08.871455 22546 hybrid_clock.cc:648] HybridClock initialized: now 1790677688871443 us; error 28 us; skew 500 ppm
Sep 29 10:28:08 dist-test-slave-9cj5 krb5kdc[22533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677688, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:08.878451 22546 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:28:08.879436 22546 webserver.cc:460] Webserver started at http://127.20.92.62:39581/ using document root <none> and password file <none>
I20260929 10:28:08.880023 22546 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:08.880084 22546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:08.880307 22546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:08.882506 22546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "ac1e7544669a4d08bb861ec2c8cfd64a"
format_stamp: "Formatted at 2026-09-29 10:28:08 on dist-test-slave-9cj5"
server_key: "f33029504f28479f6aa728b30074fb73"
server_key_iv: "afdec636da8bbf309953d59e48ac832b"
server_key_version: "encryptionkey@0"
I20260929 10:28:08.883085 22546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "ac1e7544669a4d08bb861ec2c8cfd64a"
format_stamp: "Formatted at 2026-09-29 10:28:08 on dist-test-slave-9cj5"
server_key: "f33029504f28479f6aa728b30074fb73"
server_key_iv: "afdec636da8bbf309953d59e48ac832b"
server_key_version: "encryptionkey@0"
I20260929 10:28:08.887003 22546 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 10:28:08.889587 22561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:08.890995 22546 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:28:08.891140 22546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "ac1e7544669a4d08bb861ec2c8cfd64a"
format_stamp: "Formatted at 2026-09-29 10:28:08 on dist-test-slave-9cj5"
server_key: "f33029504f28479f6aa728b30074fb73"
server_key_iv: "afdec636da8bbf309953d59e48ac832b"
server_key_version: "encryptionkey@0"
I20260929 10:28:08.891254 22546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:08.905845 22546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:08.909298 22546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:08.909541 22546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:08.917532 22546 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:38219
I20260929 10:28:08.917557 22623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:38219 every 8 connection(s)
I20260929 10:28:08.918557 22546 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:28:08.921450 22624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:08.927124 22624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac1e7544669a4d08bb861ec2c8cfd64a: Bootstrap starting.
I20260929 10:28:08.927989 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 22546
I20260929 10:28:08.928100 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:28:08.928386 20848 external_mini_cluster.cc:1428] Setting key d91a037a65026db5408d02992a5ed159
I20260929 10:28:08.930195 22624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac1e7544669a4d08bb861ec2c8cfd64a: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:08.930945 22624 log.cc:824] T 00000000000000000000000000000000 P ac1e7544669a4d08bb861ec2c8cfd64a: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:08.932869 22624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac1e7544669a4d08bb861ec2c8cfd64a: No bootstrap required, opened a new log
I20260929 10:28:08.936537 22624 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac1e7544669a4d08bb861ec2c8cfd64a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac1e7544669a4d08bb861ec2c8cfd64a" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38219 } }
I20260929 10:28:08.936832 22624 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac1e7544669a4d08bb861ec2c8cfd64a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:08.936928 22624 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac1e7544669a4d08bb861ec2c8cfd64a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac1e7544669a4d08bb861ec2c8cfd64a, State: Initialized, Role: FOLLOWER
Sep 29 10:28:08 dist-test-slave-9cj5 krb5kdc[22533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:08.937456 22624 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac1e7544669a4d08bb861ec2c8cfd64a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac1e7544669a4d08bb861ec2c8cfd64a" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38219 } }
I20260929 10:28:08.937623 22624 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac1e7544669a4d08bb861ec2c8cfd64a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:28:08.937692 22624 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac1e7544669a4d08bb861ec2c8cfd64a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:28:08.937819 22624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac1e7544669a4d08bb861ec2c8cfd64a [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:08.939107 22624 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac1e7544669a4d08bb861ec2c8cfd64a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac1e7544669a4d08bb861ec2c8cfd64a" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38219 } }
I20260929 10:28:08.939496 22624 leader_election.cc:302] T 00000000000000000000000000000000 P ac1e7544669a4d08bb861ec2c8cfd64a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac1e7544669a4d08bb861ec2c8cfd64a; no voters:
I20260929 10:28:08.939888 22624 leader_election.cc:288] T 00000000000000000000000000000000 P ac1e7544669a4d08bb861ec2c8cfd64a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:28:08.939966 22629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac1e7544669a4d08bb861ec2c8cfd64a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:08.940187 22629 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac1e7544669a4d08bb861ec2c8cfd64a [term 1 LEADER]: Becoming Leader. State: Replica: ac1e7544669a4d08bb861ec2c8cfd64a, State: Running, Role: LEADER
I20260929 10:28:08.940553 22629 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac1e7544669a4d08bb861ec2c8cfd64a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac1e7544669a4d08bb861ec2c8cfd64a" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38219 } }
I20260929 10:28:08.941227 22624 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac1e7544669a4d08bb861ec2c8cfd64a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:28:08.941766 22629 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac1e7544669a4d08bb861ec2c8cfd64a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac1e7544669a4d08bb861ec2c8cfd64a. Latest consensus state: current_term: 1 leader_uuid: "ac1e7544669a4d08bb861ec2c8cfd64a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac1e7544669a4d08bb861ec2c8cfd64a" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38219 } } }
I20260929 10:28:08.941881 22629 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac1e7544669a4d08bb861ec2c8cfd64a [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:08.942543 22634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:28:08.942572 22630 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac1e7544669a4d08bb861ec2c8cfd64a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac1e7544669a4d08bb861ec2c8cfd64a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac1e7544669a4d08bb861ec2c8cfd64a" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38219 } } }
I20260929 10:28:08.942781 22630 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac1e7544669a4d08bb861ec2c8cfd64a [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:08.944329 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:08.929608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45680 (local address 127.20.92.62:38219)
0929 10:28:08.930112 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:08.930127 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:08.930168 (+ 41us) server_negotiation.cc:407] Connection header received
0929 10:28:08.930952 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:08.930970 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:08.931361 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:08.931701 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:08.932598 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:08.933618 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:08.934511 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:08.934993 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:08.935174 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:08.938114 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:08.938121 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:08.938142 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:08.938175 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:08.940409 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:08.940968 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:08.940974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:08.940980 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:08.941126 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:08.941432 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:08.941435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:08.941437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:08.942111 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:08.942294 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:08.943072 (+ 778us) server_negotiation.cc:299] Negotiation successful
0929 10:28:08.943378 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":167,"threads_started":1}
I20260929 10:28:08.946110 22634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:28:08.952162 22634 catalog_manager.cc:1347] Generated new cluster ID: cfcc758c6c8d4b6db483f1664998f04c
I20260929 10:28:08.952239 22634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:28:08.970867 22634 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:28:08.972405 22634 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:28:08.982910 22634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac1e7544669a4d08bb861ec2c8cfd64a: Generated new TSK 0
I20260929 10:28:08.983893 22634 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
I20260929 10:28:09.021523 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:0
--local_ip_for_outbound_sockets=127.20.92.1
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:38219
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:40617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:09.140208 22650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:09.140516 22650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:09.140569 22650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:09.145485 22650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:09.145565 22650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:09.145617 22650 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 10:28:09.145675 22650 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 10:28:09.145732 22650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:28:09.152005 22650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:40617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.92.62:38219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:09.153173 22650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:09.154292 22650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:28:09.164475 22650 server_base.cc:1034] running on GCE node
W20260929 10:28:09.164424 22655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:09.164424 22656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:09.164458 22658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:09.165158 22650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:09.166009 22650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:09.167256 22650 hybrid_clock.cc:648] HybridClock initialized: now 1790677689167218 us; error 46 us; skew 500 ppm
Sep 29 10:28:09 dist-test-slave-9cj5 krb5kdc[22533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677689, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:09.170476 22650 init.cc:377] Logged in from keytab as kudu/127.20.92.1@KRBTEST.COM (short username kudu)
I20260929 10:28:09.171659 22650 webserver.cc:460] Webserver started at http://127.20.92.1:40013/ using document root <none> and password file <none>
I20260929 10:28:09.172297 22650 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:09.172400 22650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:09.172667 22650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:09.175130 22650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-0/data/instance:
uuid: "7743196215194f4b93a5abdf914805b2"
format_stamp: "Formatted at 2026-09-29 10:28:09 on dist-test-slave-9cj5"
server_key: "d4b2b97b4d83c23b0cfbd94cb9af3f5a"
server_key_iv: "81f3d8cea33f10f1336f8062108e08aa"
server_key_version: "encryptionkey@0"
I20260929 10:28:09.175793 22650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance:
uuid: "7743196215194f4b93a5abdf914805b2"
format_stamp: "Formatted at 2026-09-29 10:28:09 on dist-test-slave-9cj5"
server_key: "d4b2b97b4d83c23b0cfbd94cb9af3f5a"
server_key_iv: "81f3d8cea33f10f1336f8062108e08aa"
server_key_version: "encryptionkey@0"
I20260929 10:28:09.180104 22650 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:28:09.183051 22665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:09.184405 22650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:28:09.184567 22650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "7743196215194f4b93a5abdf914805b2"
format_stamp: "Formatted at 2026-09-29 10:28:09 on dist-test-slave-9cj5"
server_key: "d4b2b97b4d83c23b0cfbd94cb9af3f5a"
server_key_iv: "81f3d8cea33f10f1336f8062108e08aa"
server_key_version: "encryptionkey@0"
I20260929 10:28:09.184708 22650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:09.224362 22650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:09.227866 22650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:09.228188 22650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:09.229725 22650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:09.229846 22650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:09.229893 22650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:09.229947 22650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 10:28:09 dist-test-slave-9cj5 krb5kdc[22533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677689, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:09.241400 22650 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:44555
I20260929 10:28:09.241621 22780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:44555 every 8 connection(s)
I20260929 10:28:09.242385 22650 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
I20260929 10:28:09.244488 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:09.230783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:50249 (local address 127.20.92.62:38219)
0929 10:28:09.230921 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:09.230924 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:09.232129 (+ 1205us) server_negotiation.cc:407] Connection header received
0929 10:28:09.233355 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:09.233358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:09.233420 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:09.233534 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:09.234925 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.235502 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.236786 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.237063 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.237195 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:09.240343 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:09.240347 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:09.240350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:09.240385 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:09.242167 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:09.242797 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:09.242801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:09.242803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:09.242935 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:09.243356 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:09.243360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:09.243361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:09.243619 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:09.243715 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:09.244236 (+ 521us) server_negotiation.cc:299] Negotiation successful
0929 10:28:09.244364 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 10:28:09.245329 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:09.231364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38219 (local address 127.20.92.1:50249)
0929 10:28:09.231957 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:09.232000 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:09.233107 (+ 1107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:09.233723 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:09.233730 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:09.234179 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:09.234740 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:09.234763 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.235644 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.235647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:09.236638 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:09.236648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.237241 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.237245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:09.237471 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:09.238207 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:09.238254 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:09.240151 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:09.242307 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:09.242314 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:09.242329 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:09.242690 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:09.243059 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:09.243063 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:09.243065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:09.243226 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:09.243777 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:09.243782 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:09.244009 (+ 227us) client_negotiation.cc:769] Sending connection context
0929 10:28:09.244160 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 10:28:09.244422 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":143,"threads_started":1}
I20260929 10:28:09.246754 22781 heartbeater.cc:347] Connected to a master server at 127.20.92.62:38219
I20260929 10:28:09.246935 22781 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:09.247526 22781 heartbeater.cc:511] Master 127.20.92.62:38219 requested a full tablet report, sending...
I20260929 10:28:09.248766 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 22650
I20260929 10:28:09.248896 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance
I20260929 10:28:09.249231 20848 external_mini_cluster.cc:1428] Setting key fe98935167a9e81126d1f36693851570
I20260929 10:28:09.249279 22576 ts_manager.cc:194] Registered new tserver with Master: 7743196215194f4b93a5abdf914805b2 (127.20.92.1:44555)
I20260929 10:28:09.250810 22576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.1@KRBTEST.COM'} at 127.20.92.1:50249
I20260929 10:28:09.257471 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:09.249667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:60187 (local address 127.20.92.62:38219)
0929 10:28:09.249807 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:09.249811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:09.249824 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:28:09.249859 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:09.249862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:09.249901 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:09.249979 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:09.250498 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.251217 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.252173 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.252540 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.252660 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:09.254032 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:09.254035 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:09.254037 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:09.254061 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:09.255553 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:09.256103 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:09.256106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:09.256107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:09.256187 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:09.256586 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:09.256589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:09.256591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:09.256785 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:09.256878 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:09.257218 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 10:28:09.257350 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 10:28:09.257566 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:09.249143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38219 (local address 127.20.92.1:60187)
0929 10:28:09.249481 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:09.249497 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:09.249588 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:09.250002 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:09.250005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:09.250231 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:09.250372 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:09.250380 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.251368 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.251371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:09.252034 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:09.252043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.252671 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.252675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:09.252769 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:09.253370 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:09.253385 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:09.253890 (+ 505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:09.255675 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:09.255680 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:09.255683 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:09.255972 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:09.256294 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:09.256300 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:09.256304 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:09.256451 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:09.256894 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:09.256898 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:09.257017 (+ 119us) client_negotiation.cc:769] Sending connection context
0929 10:28:09.257138 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 10:28:09.257313 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for kudu/127.20.92.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
I20260929 10:28:09.281620 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:0
--local_ip_for_outbound_sockets=127.20.92.2
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:38219
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:40617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:09.396544 22790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:09.396857 22790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:09.396955 22790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:09.401616 22790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:09.401700 22790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:09.401803 22790 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 10:28:09.401835 22790 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 10:28:09.401889 22790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:28:09.407516 22790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:40617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.92.62:38219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:09.408618 22790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:09.409615 22790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:09.420125 22796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:09.420147 22798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:09.420210 22790 server_base.cc:1034] running on GCE node
W20260929 10:28:09.420125 22795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:09.420861 22790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:09.421571 22790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:09.422760 22790 hybrid_clock.cc:648] HybridClock initialized: now 1790677689422730 us; error 56 us; skew 500 ppm
Sep 29 10:28:09 dist-test-slave-9cj5 krb5kdc[22533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677689, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:09.425685 22790 init.cc:377] Logged in from keytab as kudu/127.20.92.2@KRBTEST.COM (short username kudu)
I20260929 10:28:09.426724 22790 webserver.cc:460] Webserver started at http://127.20.92.2:38897/ using document root <none> and password file <none>
I20260929 10:28:09.427310 22790 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:09.427439 22790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:09.427668 22790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:09.429857 22790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-1/data/instance:
uuid: "bb810c0459974dc6a133b30479c09c91"
format_stamp: "Formatted at 2026-09-29 10:28:09 on dist-test-slave-9cj5"
server_key: "8abf2e916552e0e3ac1f5fa081329def"
server_key_iv: "896f9c3621daedef8b11661730ad0c68"
server_key_version: "encryptionkey@0"
I20260929 10:28:09.430481 22790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance:
uuid: "bb810c0459974dc6a133b30479c09c91"
format_stamp: "Formatted at 2026-09-29 10:28:09 on dist-test-slave-9cj5"
server_key: "8abf2e916552e0e3ac1f5fa081329def"
server_key_iv: "896f9c3621daedef8b11661730ad0c68"
server_key_version: "encryptionkey@0"
I20260929 10:28:09.434444 22790 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 10:28:09.437063 22805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:09.438431 22790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 10:28:09.438575 22790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "bb810c0459974dc6a133b30479c09c91"
format_stamp: "Formatted at 2026-09-29 10:28:09 on dist-test-slave-9cj5"
server_key: "8abf2e916552e0e3ac1f5fa081329def"
server_key_iv: "896f9c3621daedef8b11661730ad0c68"
server_key_version: "encryptionkey@0"
I20260929 10:28:09.438699 22790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:09.453227 22790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:09.456302 22790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:09.456517 22790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:09.458021 22790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:09.458098 22790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:09.458166 22790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:09.458220 22790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 10:28:09 dist-test-slave-9cj5 krb5kdc[22533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677689, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:09.469605 22790 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:41717
I20260929 10:28:09.469677 22920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:41717 every 8 connection(s)
I20260929 10:28:09.470722 22790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
I20260929 10:28:09.472162 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:09.459061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:55029 (local address 127.20.92.62:38219)
0929 10:28:09.459257 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:09.459262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:09.460207 (+ 945us) server_negotiation.cc:407] Connection header received
0929 10:28:09.461202 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:09.461208 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:09.461277 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:09.461340 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:09.462871 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.463453 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.464593 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.464914 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.465044 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:09.468034 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:09.468038 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:09.468041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:09.468076 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:09.469951 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:09.470429 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:09.470432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:09.470434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:09.470522 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:09.470910 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:09.470913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:09.470915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:09.471174 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:09.471269 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:09.471931 (+ 662us) server_negotiation.cc:299] Negotiation successful
0929 10:28:09.472044 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 10:28:09.472918 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:09.459497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38219 (local address 127.20.92.2:55029)
0929 10:28:09.460040 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:09.460084 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:09.460937 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:09.461718 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:09.461725 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:09.462188 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:09.462721 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:09.462738 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.463584 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.463587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:09.464464 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:09.464473 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.465096 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.465100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:09.465350 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:09.466045 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:09.466070 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:09.467859 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:09.470062 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:09.470066 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:09.470077 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:09.470318 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:09.470607 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:09.470609 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:09.470610 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:09.470739 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:09.471332 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:09.471340 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:09.471633 (+ 293us) client_negotiation.cc:769] Sending connection context
0929 10:28:09.471826 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 10:28:09.472082 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":141,"threads_started":1}
I20260929 10:28:09.474114 22921 heartbeater.cc:347] Connected to a master server at 127.20.92.62:38219
I20260929 10:28:09.474277 22921 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:09.474768 22921 heartbeater.cc:511] Master 127.20.92.62:38219 requested a full tablet report, sending...
I20260929 10:28:09.475886 22578 ts_manager.cc:194] Registered new tserver with Master: bb810c0459974dc6a133b30479c09c91 (127.20.92.2:41717)
I20260929 10:28:09.476660 22578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.2@KRBTEST.COM'} at 127.20.92.2:55029
I20260929 10:28:09.478579 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 22790
I20260929 10:28:09.478688 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance
I20260929 10:28:09.478966 20848 external_mini_cluster.cc:1428] Setting key a09504bb4f78cac98635758aab18b7c5
I20260929 10:28:09.484217 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:09.476248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:39639 (local address 127.20.92.62:38219)
0929 10:28:09.476381 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:09.476386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:09.476572 (+ 186us) server_negotiation.cc:407] Connection header received
0929 10:28:09.476677 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:09.476679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:09.476716 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:09.476808 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:09.477433 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.478202 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.479378 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.479706 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.479774 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:09.480955 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:09.480962 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:09.480968 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:09.480999 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:09.482562 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:09.483006 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:09.483010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:09.483012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:09.483099 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:09.483379 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:09.483383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:09.483385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:09.483592 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:09.483701 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:09.483957 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 10:28:09.484061 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 10:28:09.484226 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:09.476191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38219 (local address 127.20.92.2:39639)
0929 10:28:09.476487 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:09.476501 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:09.476595 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:09.476947 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:09.476969 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:09.477190 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:09.477313 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:09.477320 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.478343 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.478349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:09.479234 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:09.479244 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.479903 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.479905 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:09.479972 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:09.480503 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:09.480514 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:09.480821 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:09.482689 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:09.482693 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:09.482695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:09.482903 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:09.483191 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:09.483196 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:09.483198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:09.483283 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:09.483690 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:09.483693 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:09.483791 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 10:28:09.483906 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 10:28:09.484054 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":141,"threads_started":1}
WARNING: no policy specified for kudu/127.20.92.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
I20260929 10:28:09.510605 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:0
--local_ip_for_outbound_sockets=127.20.92.3
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:38219
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:40617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:09.625306 22930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:09.625588 22930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:09.625638 22930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:09.629935 22930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:09.630012 22930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:09.630055 22930 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 10:28:09.630070 22930 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 10:28:09.630105 22930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:28:09.635540 22930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:40617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.92.62:38219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:09.636590 22930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:09.637532 22930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:09.647547 22936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:09.647538 22938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:09.647542 22935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:09.648092 22930 server_base.cc:1034] running on GCE node
I20260929 10:28:09.648499 22930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:09.649196 22930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:09.650398 22930 hybrid_clock.cc:648] HybridClock initialized: now 1790677689650381 us; error 30 us; skew 500 ppm
Sep 29 10:28:09 dist-test-slave-9cj5 krb5kdc[22533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677689, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:09.653225 22930 init.cc:377] Logged in from keytab as kudu/127.20.92.3@KRBTEST.COM (short username kudu)
I20260929 10:28:09.654183 22930 webserver.cc:460] Webserver started at http://127.20.92.3:35729/ using document root <none> and password file <none>
I20260929 10:28:09.654752 22930 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:09.654841 22930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:09.655076 22930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:09.657196 22930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-2/data/instance:
uuid: "951cf90099f0458e9b85d9e47a6ed921"
format_stamp: "Formatted at 2026-09-29 10:28:09 on dist-test-slave-9cj5"
server_key: "20000749dacc3c6ebc05d9f50d2b87b8"
server_key_iv: "dea962baf1ce6e8b52721421cb580ae7"
server_key_version: "encryptionkey@0"
I20260929 10:28:09.657858 22930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance:
uuid: "951cf90099f0458e9b85d9e47a6ed921"
format_stamp: "Formatted at 2026-09-29 10:28:09 on dist-test-slave-9cj5"
server_key: "20000749dacc3c6ebc05d9f50d2b87b8"
server_key_iv: "dea962baf1ce6e8b52721421cb580ae7"
server_key_version: "encryptionkey@0"
I20260929 10:28:09.661715 22930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 10:28:09.664537 22945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:09.666077 22930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:28:09.666220 22930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "951cf90099f0458e9b85d9e47a6ed921"
format_stamp: "Formatted at 2026-09-29 10:28:09 on dist-test-slave-9cj5"
server_key: "20000749dacc3c6ebc05d9f50d2b87b8"
server_key_iv: "dea962baf1ce6e8b52721421cb580ae7"
server_key_version: "encryptionkey@0"
I20260929 10:28:09.666363 22930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:09.681668 22930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:09.685081 22930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:09.685304 22930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:09.686813 22930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:09.686897 22930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:09.686975 22930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:09.687018 22930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:09.697957 22930 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:40001
I20260929 10:28:09.698014 23060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:40001 every 8 connection(s)
Sep 29 10:28:09 dist-test-slave-9cj5 krb5kdc[22533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677689, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:09.698966 22930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:28:09.702775 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:09.687708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:45217 (local address 127.20.92.62:38219)
0929 10:28:09.687838 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:09.687842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:09.689106 (+ 1264us) server_negotiation.cc:407] Connection header received
0929 10:28:09.690683 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:09.690688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:09.690760 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:09.690860 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:09.692694 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.693624 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.695444 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.695830 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.695974 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:09.698884 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:09.698891 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:09.698895 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:09.698928 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:09.700585 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:09.701187 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:09.701190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:09.701191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:09.701284 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:09.701618 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:09.701620 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:09.701622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:09.701924 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:09.702020 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:09.702521 (+ 501us) server_negotiation.cc:299] Negotiation successful
0929 10:28:09.702634 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 10:28:09.703577 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:09.688272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38219 (local address 127.20.92.3:45217)
0929 10:28:09.688916 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:09.688961 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:09.690366 (+ 1405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:09.691194 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:09.691203 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:09.691732 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:09.692490 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:09.692514 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.693863 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.693874 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:09.695311 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:09.695321 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.696012 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.696017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:09.696227 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:09.696982 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:09.696998 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:09.698685 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:09.700682 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:09.700686 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:09.700696 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:09.701016 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:09.701382 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:09.701385 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:09.701386 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:09.701520 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:09.702035 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:09.702040 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:09.702274 (+ 234us) client_negotiation.cc:769] Sending connection context
0929 10:28:09.702429 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 10:28:09.702681 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":145,"threads_started":1}
I20260929 10:28:09.704980 23061 heartbeater.cc:347] Connected to a master server at 127.20.92.62:38219
I20260929 10:28:09.705132 23061 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:09.705715 23061 heartbeater.cc:511] Master 127.20.92.62:38219 requested a full tablet report, sending...
I20260929 10:28:09.706957 22578 ts_manager.cc:194] Registered new tserver with Master: 951cf90099f0458e9b85d9e47a6ed921 (127.20.92.3:40001)
I20260929 10:28:09.707849 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 22930
I20260929 10:28:09.707939 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance
I20260929 10:28:09.707921 22578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.3@KRBTEST.COM'} at 127.20.92.3:45217
I20260929 10:28:09.708307 20848 external_mini_cluster.cc:1428] Setting key 0a2a2d63f0e61644962ff3df2701ad92
I20260929 10:28:09.710589 20848 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:28:09.716750 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:09.707465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38219 (local address 127.20.92.3:40307)
0929 10:28:09.708610 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:09.708627 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:09.708749 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:09.709005 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:09.709008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:09.709365 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:09.709500 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:09.709508 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.710511 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.710515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:09.711372 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:09.711384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.712071 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.712078 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:09.712213 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:09.712834 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:09.712853 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:09.713162 (+ 309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:09.715160 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:09.715166 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:09.715169 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:09.715436 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:09.715707 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:09.715710 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:09.715712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:09.715808 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:09.716307 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:09.716309 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:09.716396 (+ 87us) client_negotiation.cc:769] Sending connection context
0929 10:28:09.716484 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 10:28:09.716603 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1031,"mutex_wait_us":1,"thread_start_us":96,"threads_started":1}
I20260929 10:28:09.716951 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:09.707536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:40307 (local address 127.20.92.62:38219)
0929 10:28:09.707656 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:09.707663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:09.708727 (+ 1064us) server_negotiation.cc:407] Connection header received
0929 10:28:09.708851 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:09.708854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:09.708902 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:09.708988 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:09.709622 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.710387 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.711507 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.711935 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.712057 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:09.713308 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:09.713312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:09.713315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:09.713355 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:09.715030 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:09.715537 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:09.715539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:09.715541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:09.715612 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:09.715921 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:09.715924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:09.715926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:09.716195 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:09.716318 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:09.716576 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 10:28:09.716716 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 10:28:09 dist-test-slave-9cj5 krb5kdc[22533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 10:28:09 dist-test-slave-9cj5 krb5kdc[22533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:09.738444 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:09.729711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45692 (local address 127.20.92.62:38219)
0929 10:28:09.729894 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:09.729898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:09.729908 (+ 10us) server_negotiation.cc:407] Connection header received
0929 10:28:09.729949 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:09.729951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:09.729988 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:09.730086 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:09.730790 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.731322 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.732218 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.732446 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.732559 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:09.735207 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:09.735210 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:09.735212 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:09.735237 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:09.736714 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:09.737106 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:09.737108 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:09.737109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:09.737182 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:09.737513 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:09.737516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:09.737517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:09.737845 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:09.737970 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:09.738212 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 10:28:09.738313 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260929 10:28:09.741600 22578 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:28:09.745473 22578 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:28:09.760994 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:09.754681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:41717 (local address 127.0.0.1:58410)
0929 10:28:09.755468 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:09.755523 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:09.755667 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:09.756611 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:09.756617 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:09.756641 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:09.756794 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:09.756800 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.758008 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.758013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:09.759415 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:09.759425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.760395 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.760399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:09.760539 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:09.760591 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 10:28:09.760694 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 10:28:09.760828 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":30,"thread_start_us":102,"threads_started":1}
I20260929 10:28:09.761066 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:09.755885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39858 (local address 127.20.92.3:40001)
0929 10:28:09.756198 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:09.756211 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:09.756234 (+ 23us) server_negotiation.cc:407] Connection header received
0929 10:28:09.756312 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:09.756317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:09.756508 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:09.756720 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:09.756922 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.758163 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.759529 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.760238 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.760373 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:09.760563 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:09.760780 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 10:28:09.760884 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":127,"threads_started":1}
I20260929 10:28:09.761286 23079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:09.754792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:40001 (local address 127.0.0.1:39858)
0929 10:28:09.755572 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:09.755588 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:09.755676 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:09.756694 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:09.756697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:09.756710 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:09.756794 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:09.756800 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.758288 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.758291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:09.759425 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:09.759432 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.760371 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.760375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:09.760579 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:09.760613 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 10:28:09.761017 (+ 404us) client_negotiation.cc:240] Negotiation successful
0929 10:28:09.761111 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":51,"thread_start_us":111,"threads_started":1}
I20260929 10:28:09.761698 23078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:09.755064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:44555 (local address 127.0.0.1:34322)
0929 10:28:09.755881 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:09.755891 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:09.755947 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:09.757136 (+ 1189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:09.757139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:09.757158 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:09.757268 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:09.757276 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.758580 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.758587 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:09.759820 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:09.759832 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.760998 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.761003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:09.761116 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:09.761147 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 10:28:09.761456 (+ 309us) client_negotiation.cc:240] Negotiation successful
0929 10:28:09.761536 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"mutex_wait_us":99,"thread_start_us":97,"threads_started":1}
I20260929 10:28:09.761873 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:09.755499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34322 (local address 127.20.92.1:44555)
0929 10:28:09.756543 (+ 1044us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:09.756550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:09.756575 (+ 25us) server_negotiation.cc:407] Connection header received
0929 10:28:09.756668 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:09.756673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:09.756833 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:09.756986 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:09.757419 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.758466 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.759960 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.760728 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.760820 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:09.760939 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:09.761540 (+ 601us) server_negotiation.cc:299] Negotiation successful
0929 10:28:09.761655 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":942,"thread_start_us":96,"threads_started":1}
I20260929 10:28:09.762498 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:09.754792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58410 (local address 127.20.92.2:41717)
0929 10:28:09.756142 (+ 1350us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:09.756150 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:09.756170 (+ 20us) server_negotiation.cc:407] Connection header received
0929 10:28:09.756240 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:09.756243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:09.756442 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:09.756950 (+ 508us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:09.757066 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.757839 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.759569 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.760261 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.762067 (+ 1806us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:09.762242 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:09.762288 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 10:28:09.762368 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1260,"thread_start_us":123,"threads_started":1}
I20260929 10:28:09.764485 22715 tablet_service.cc:1467] Processing CreateTablet for tablet 38234e2f56c945f1b53788b724512df3 (DEFAULT_TABLE table=test-table [id=f5e11d7eb89749d8a6f288fa16771d3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:09.764536 22995 tablet_service.cc:1467] Processing CreateTablet for tablet 38234e2f56c945f1b53788b724512df3 (DEFAULT_TABLE table=test-table [id=f5e11d7eb89749d8a6f288fa16771d3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:09.764958 22855 tablet_service.cc:1467] Processing CreateTablet for tablet 38234e2f56c945f1b53788b724512df3 (DEFAULT_TABLE table=test-table [id=f5e11d7eb89749d8a6f288fa16771d3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:09.765537 22715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38234e2f56c945f1b53788b724512df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:09.765556 22995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38234e2f56c945f1b53788b724512df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:09.765848 22855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38234e2f56c945f1b53788b724512df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:09.772810 23084 tablet_bootstrap.cc:492] T 38234e2f56c945f1b53788b724512df3 P 7743196215194f4b93a5abdf914805b2: Bootstrap starting.
I20260929 10:28:09.773334 23085 tablet_bootstrap.cc:492] T 38234e2f56c945f1b53788b724512df3 P bb810c0459974dc6a133b30479c09c91: Bootstrap starting.
I20260929 10:28:09.773890 23086 tablet_bootstrap.cc:492] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921: Bootstrap starting.
I20260929 10:28:09.775270 23084 tablet_bootstrap.cc:654] T 38234e2f56c945f1b53788b724512df3 P 7743196215194f4b93a5abdf914805b2: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:09.776000 23086 tablet_bootstrap.cc:654] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:09.776129 23084 log.cc:824] T 38234e2f56c945f1b53788b724512df3 P 7743196215194f4b93a5abdf914805b2: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:09.776784 23086 log.cc:824] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:09.778258 23084 tablet_bootstrap.cc:492] T 38234e2f56c945f1b53788b724512df3 P 7743196215194f4b93a5abdf914805b2: No bootstrap required, opened a new log
I20260929 10:28:09.778515 23084 ts_tablet_manager.cc:1397] T 38234e2f56c945f1b53788b724512df3 P 7743196215194f4b93a5abdf914805b2: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260929 10:28:09.779035 23085 tablet_bootstrap.cc:654] T 38234e2f56c945f1b53788b724512df3 P bb810c0459974dc6a133b30479c09c91: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:09.779681 23085 log.cc:824] T 38234e2f56c945f1b53788b724512df3 P bb810c0459974dc6a133b30479c09c91: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:09.781152 23086 tablet_bootstrap.cc:492] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921: No bootstrap required, opened a new log
I20260929 10:28:09.781360 23086 ts_tablet_manager.cc:1397] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260929 10:28:09.781570 23085 tablet_bootstrap.cc:492] T 38234e2f56c945f1b53788b724512df3 P bb810c0459974dc6a133b30479c09c91: No bootstrap required, opened a new log
I20260929 10:28:09.781781 23085 ts_tablet_manager.cc:1397] T 38234e2f56c945f1b53788b724512df3 P bb810c0459974dc6a133b30479c09c91: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20260929 10:28:09.782797 23084 raft_consensus.cc:348] T 38234e2f56c945f1b53788b724512df3 P 7743196215194f4b93a5abdf914805b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } } peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } }
I20260929 10:28:09.783098 23084 raft_consensus.cc:374] T 38234e2f56c945f1b53788b724512df3 P 7743196215194f4b93a5abdf914805b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:09.783154 23084 raft_consensus.cc:729] T 38234e2f56c945f1b53788b724512df3 P 7743196215194f4b93a5abdf914805b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7743196215194f4b93a5abdf914805b2, State: Initialized, Role: FOLLOWER
I20260929 10:28:09.783715 23084 consensus_queue.cc:260] T 38234e2f56c945f1b53788b724512df3 P 7743196215194f4b93a5abdf914805b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } } peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } }
I20260929 10:28:09.784482 22781 heartbeater.cc:503] Master 127.20.92.62:38219 was elected leader, sending a full tablet report...
I20260929 10:28:09.784731 23084 ts_tablet_manager.cc:1428] T 38234e2f56c945f1b53788b724512df3 P 7743196215194f4b93a5abdf914805b2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 10:28:09.785657 23085 raft_consensus.cc:348] T 38234e2f56c945f1b53788b724512df3 P bb810c0459974dc6a133b30479c09c91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } } peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } }
I20260929 10:28:09.785987 23085 raft_consensus.cc:374] T 38234e2f56c945f1b53788b724512df3 P bb810c0459974dc6a133b30479c09c91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:09.786046 23085 raft_consensus.cc:729] T 38234e2f56c945f1b53788b724512df3 P bb810c0459974dc6a133b30479c09c91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb810c0459974dc6a133b30479c09c91, State: Initialized, Role: FOLLOWER
I20260929 10:28:09.786409 23086 raft_consensus.cc:348] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } } peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } }
I20260929 10:28:09.786461 23085 consensus_queue.cc:260] T 38234e2f56c945f1b53788b724512df3 P bb810c0459974dc6a133b30479c09c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } } peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } }
I20260929 10:28:09.786624 23086 raft_consensus.cc:374] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:09.786677 23086 raft_consensus.cc:729] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951cf90099f0458e9b85d9e47a6ed921, State: Initialized, Role: FOLLOWER
I20260929 10:28:09.787073 23086 consensus_queue.cc:260] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } } peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } }
I20260929 10:28:09.787199 22921 heartbeater.cc:503] Master 127.20.92.62:38219 was elected leader, sending a full tablet report...
I20260929 10:28:09.787562 23085 ts_tablet_manager.cc:1428] T 38234e2f56c945f1b53788b724512df3 P bb810c0459974dc6a133b30479c09c91: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 10:28:09.788003 23086 ts_tablet_manager.cc:1428] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 10:28:09.788321 23061 heartbeater.cc:503] Master 127.20.92.62:38219 was elected leader, sending a full tablet report...
I20260929 10:28:09.859483 23092 raft_consensus.cc:482] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:28:09.859684 23092 raft_consensus.cc:504] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } } peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } }
I20260929 10:28:09.860867 23092 leader_election.cc:288] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb810c0459974dc6a133b30479c09c91 (127.20.92.2:41717), 7743196215194f4b93a5abdf914805b2 (127.20.92.1:44555)
I20260929 10:28:09.865260 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:09.861286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:57011 (local address 127.20.92.1:44555)
0929 10:28:09.861415 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:09.861420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:09.861437 (+ 17us) server_negotiation.cc:407] Connection header received
0929 10:28:09.861739 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:09.861815 (+ 76us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:09.861880 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:09.861991 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:09.862330 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.862890 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.864106 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.864639 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.864816 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:09.864886 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:09.865053 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 10:28:09.865128 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 10:28:09.865260 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:09.861176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:44555 (local address 127.20.92.3:57011)
0929 10:28:09.861297 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:09.861314 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:09.861509 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:09.862005 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:09.862009 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:09.862030 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:09.862173 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:09.862180 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.863022 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.863029 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:09.863962 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:09.863973 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.864797 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.864800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:09.864913 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:09.864938 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 10:28:09.865037 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 10:28:09.865118 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 10:28:09.866039 22735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38234e2f56c945f1b53788b724512df3" candidate_uuid: "951cf90099f0458e9b85d9e47a6ed921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7743196215194f4b93a5abdf914805b2" is_pre_election: true
I20260929 10:28:09.866364 22735 raft_consensus.cc:2454] T 38234e2f56c945f1b53788b724512df3 P 7743196215194f4b93a5abdf914805b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 951cf90099f0458e9b85d9e47a6ed921 in term 0.
I20260929 10:28:09.866600 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:09.861229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:40291 (local address 127.20.92.2:41717)
0929 10:28:09.861381 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:09.861385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:09.862198 (+ 813us) server_negotiation.cc:407] Connection header received
0929 10:28:09.862392 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:09.862396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:09.862450 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:09.862539 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:09.863472 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.864075 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.865261 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.865939 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.866080 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:09.866145 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:09.866398 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 10:28:09.866474 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 10:28:09.866632 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:09.861177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:41717 (local address 127.20.92.3:40291)
0929 10:28:09.862033 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:09.862044 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:09.862246 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:09.862551 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:09.862554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:09.862570 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:09.863150 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:09.863159 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.864225 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.864229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:09.865094 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:09.865104 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.866095 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.866099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:09.866240 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:09.866264 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 10:28:09.866381 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 10:28:09.866462 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"mutex_wait_us":425,"thread_start_us":219,"threads_started":1}
I20260929 10:28:09.866889 22947 leader_election.cc:302] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7743196215194f4b93a5abdf914805b2, 951cf90099f0458e9b85d9e47a6ed921; no voters:
I20260929 10:28:09.867188 23092 raft_consensus.cc:2781] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:28:09.867280 23092 raft_consensus.cc:482] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:28:09.867343 23092 raft_consensus.cc:3037] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:09.867374 22875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38234e2f56c945f1b53788b724512df3" candidate_uuid: "951cf90099f0458e9b85d9e47a6ed921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb810c0459974dc6a133b30479c09c91" is_pre_election: true
I20260929 10:28:09.867687 22875 raft_consensus.cc:2454] T 38234e2f56c945f1b53788b724512df3 P bb810c0459974dc6a133b30479c09c91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 951cf90099f0458e9b85d9e47a6ed921 in term 0.
I20260929 10:28:09.868494 23092 raft_consensus.cc:504] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } } peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } }
I20260929 10:28:09.868901 23092 leader_election.cc:288] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921 [CANDIDATE]: Term 1 election: Requested vote from peers bb810c0459974dc6a133b30479c09c91 (127.20.92.2:41717), 7743196215194f4b93a5abdf914805b2 (127.20.92.1:44555)
I20260929 10:28:09.869252 22735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38234e2f56c945f1b53788b724512df3" candidate_uuid: "951cf90099f0458e9b85d9e47a6ed921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7743196215194f4b93a5abdf914805b2"
I20260929 10:28:09.869393 22735 raft_consensus.cc:3037] T 38234e2f56c945f1b53788b724512df3 P 7743196215194f4b93a5abdf914805b2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:09.869414 22875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38234e2f56c945f1b53788b724512df3" candidate_uuid: "951cf90099f0458e9b85d9e47a6ed921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb810c0459974dc6a133b30479c09c91"
I20260929 10:28:09.869575 22875 raft_consensus.cc:3037] T 38234e2f56c945f1b53788b724512df3 P bb810c0459974dc6a133b30479c09c91 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:09.870775 22735 raft_consensus.cc:2454] T 38234e2f56c945f1b53788b724512df3 P 7743196215194f4b93a5abdf914805b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 951cf90099f0458e9b85d9e47a6ed921 in term 1.
I20260929 10:28:09.871119 22875 raft_consensus.cc:2454] T 38234e2f56c945f1b53788b724512df3 P bb810c0459974dc6a133b30479c09c91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 951cf90099f0458e9b85d9e47a6ed921 in term 1.
I20260929 10:28:09.871125 22947 leader_election.cc:302] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7743196215194f4b93a5abdf914805b2, 951cf90099f0458e9b85d9e47a6ed921; no voters:
I20260929 10:28:09.871507 23092 raft_consensus.cc:2781] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:09.871758 23092 raft_consensus.cc:686] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921 [term 1 LEADER]: Becoming Leader. State: Replica: 951cf90099f0458e9b85d9e47a6ed921, State: Running, Role: LEADER
I20260929 10:28:09.872152 23092 consensus_queue.cc:237] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } } peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } }
I20260929 10:28:09.875765 22578 catalog_manager.cc:5495] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921 reported cstate change: term changed from 0 to 1, leader changed from <none> to 951cf90099f0458e9b85d9e47a6ed921 (127.20.92.3). New cstate: current_term: 1 leader_uuid: "951cf90099f0458e9b85d9e47a6ed921" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } health_report { overall_health: UNKNOWN } } }
Sep 29 10:28:09 dist-test-slave-9cj5 krb5kdc[22533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677688, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:09.936990 23101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:09.927687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38219 (local address 127.0.0.1:45706)
0929 10:28:09.928128 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:09.928145 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:09.928285 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:09.928546 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:09.928549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:09.929038 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:09.929148 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:09.929156 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.930028 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.930031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:09.930941 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:09.930950 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.931370 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.931374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:09.931458 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:09.932049 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:09.932065 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:09.933624 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:09.935266 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:09.935278 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:09.935285 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:09.935525 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:09.935822 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:09.935824 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:09.935826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:09.935908 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:09.936293 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:09.936302 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:09.936437 (+ 135us) client_negotiation.cc:769] Sending connection context
0929 10:28:09.936554 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 10:28:09.936745 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":135,"threads_started":1}
I20260929 10:28:09.936988 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:09.927772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45706 (local address 127.20.92.62:38219)
0929 10:28:09.927914 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:09.927920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:09.928261 (+ 341us) server_negotiation.cc:407] Connection header received
0929 10:28:09.928372 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:09.928374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:09.928416 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:09.928530 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:09.929298 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.929925 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.931056 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:09.931260 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:09.931335 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:09.933803 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:09.933807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:09.933809 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:09.933829 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:09.935152 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:09.935626 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:09.935633 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:09.935635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:09.935721 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:09.936003 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:09.936007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:09.936008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:09.936185 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:09.936293 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:09.936609 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 10:28:09.936812 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":40}
I20260929 10:28:09.941221 22578 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:45706:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 10:28:09.942143 22578 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:28:09.947772 22715 tablet_service.cc:1467] Processing CreateTablet for tablet bbc9b98b3b2646fba9200e4c1607ea5a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c43deff669c64d76a4da158ca8794fa7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 10:28:09.947770 22855 tablet_service.cc:1467] Processing CreateTablet for tablet bbc9b98b3b2646fba9200e4c1607ea5a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c43deff669c64d76a4da158ca8794fa7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 10:28:09.948038 22995 tablet_service.cc:1467] Processing CreateTablet for tablet bbc9b98b3b2646fba9200e4c1607ea5a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c43deff669c64d76a4da158ca8794fa7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 10:28:09.948547 22715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbc9b98b3b2646fba9200e4c1607ea5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:09.948547 22855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbc9b98b3b2646fba9200e4c1607ea5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:09.948660 22995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbc9b98b3b2646fba9200e4c1607ea5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:09.951320 23086 tablet_bootstrap.cc:492] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921: Bootstrap starting.
W20260929 10:28:09.951556 23062 tablet.cc:2369] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:28:09.951730 23085 tablet_bootstrap.cc:492] T bbc9b98b3b2646fba9200e4c1607ea5a P bb810c0459974dc6a133b30479c09c91: Bootstrap starting.
I20260929 10:28:09.952181 23084 tablet_bootstrap.cc:492] T bbc9b98b3b2646fba9200e4c1607ea5a P 7743196215194f4b93a5abdf914805b2: Bootstrap starting.
I20260929 10:28:09.952353 23086 tablet_bootstrap.cc:654] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:09.952678 23085 tablet_bootstrap.cc:654] T bbc9b98b3b2646fba9200e4c1607ea5a P bb810c0459974dc6a133b30479c09c91: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:09.953235 23084 tablet_bootstrap.cc:654] T bbc9b98b3b2646fba9200e4c1607ea5a P 7743196215194f4b93a5abdf914805b2: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:09.953998 23085 tablet_bootstrap.cc:492] T bbc9b98b3b2646fba9200e4c1607ea5a P bb810c0459974dc6a133b30479c09c91: No bootstrap required, opened a new log
I20260929 10:28:09.954066 23085 ts_tablet_manager.cc:1397] T bbc9b98b3b2646fba9200e4c1607ea5a P bb810c0459974dc6a133b30479c09c91: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260929 10:28:09.954463 23084 tablet_bootstrap.cc:492] T bbc9b98b3b2646fba9200e4c1607ea5a P 7743196215194f4b93a5abdf914805b2: No bootstrap required, opened a new log
I20260929 10:28:09.954535 23084 ts_tablet_manager.cc:1397] T bbc9b98b3b2646fba9200e4c1607ea5a P 7743196215194f4b93a5abdf914805b2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 10:28:09.954484 23085 raft_consensus.cc:348] T bbc9b98b3b2646fba9200e4c1607ea5a P bb810c0459974dc6a133b30479c09c91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:09.954689 23085 raft_consensus.cc:374] T bbc9b98b3b2646fba9200e4c1607ea5a P bb810c0459974dc6a133b30479c09c91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:09.954734 23085 raft_consensus.cc:729] T bbc9b98b3b2646fba9200e4c1607ea5a P bb810c0459974dc6a133b30479c09c91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb810c0459974dc6a133b30479c09c91, State: Initialized, Role: FOLLOWER
I20260929 10:28:09.954882 23085 consensus_queue.cc:260] T bbc9b98b3b2646fba9200e4c1607ea5a P bb810c0459974dc6a133b30479c09c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:09.955024 23084 raft_consensus.cc:348] T bbc9b98b3b2646fba9200e4c1607ea5a P 7743196215194f4b93a5abdf914805b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:09.955147 23084 raft_consensus.cc:374] T bbc9b98b3b2646fba9200e4c1607ea5a P 7743196215194f4b93a5abdf914805b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:09.955171 23084 raft_consensus.cc:729] T bbc9b98b3b2646fba9200e4c1607ea5a P 7743196215194f4b93a5abdf914805b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7743196215194f4b93a5abdf914805b2, State: Initialized, Role: FOLLOWER
I20260929 10:28:09.955170 23085 ts_tablet_manager.cc:1428] T bbc9b98b3b2646fba9200e4c1607ea5a P bb810c0459974dc6a133b30479c09c91: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 10:28:09.955140 23091 tablet_replica.cc:439] T bbc9b98b3b2646fba9200e4c1607ea5a P bb810c0459974dc6a133b30479c09c91: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } } }
I20260929 10:28:09.955267 23091 tablet_replica.cc:442] T bbc9b98b3b2646fba9200e4c1607ea5a P bb810c0459974dc6a133b30479c09c91: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:28:09.955291 23084 consensus_queue.cc:260] T bbc9b98b3b2646fba9200e4c1607ea5a P 7743196215194f4b93a5abdf914805b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:09.955562 23084 ts_tablet_manager.cc:1428] T bbc9b98b3b2646fba9200e4c1607ea5a P 7743196215194f4b93a5abdf914805b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 10:28:09.955529 23090 tablet_replica.cc:439] T bbc9b98b3b2646fba9200e4c1607ea5a P 7743196215194f4b93a5abdf914805b2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } } }
I20260929 10:28:09.955664 23090 tablet_replica.cc:442] T bbc9b98b3b2646fba9200e4c1607ea5a P 7743196215194f4b93a5abdf914805b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:28:09.956656 23086 tablet_bootstrap.cc:492] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921: No bootstrap required, opened a new log
I20260929 10:28:09.956768 23086 ts_tablet_manager.cc:1397] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260929 10:28:09.957248 23086 raft_consensus.cc:348] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:09.957386 23086 raft_consensus.cc:374] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:09.957429 23086 raft_consensus.cc:729] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951cf90099f0458e9b85d9e47a6ed921, State: Initialized, Role: FOLLOWER
I20260929 10:28:09.957551 23086 consensus_queue.cc:260] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:09.957850 23086 ts_tablet_manager.cc:1428] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 10:28:09.957816 23092 tablet_replica.cc:439] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } } }
I20260929 10:28:09.957935 23092 tablet_replica.cc:442] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921: This TxnStatusTablet replica's current role is: FOLLOWER
W20260929 10:28:09.971930 22922 tablet.cc:2369] T 38234e2f56c945f1b53788b724512df3 P bb810c0459974dc6a133b30479c09c91: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:28:09.994156 22782 tablet.cc:2369] T 38234e2f56c945f1b53788b724512df3 P 7743196215194f4b93a5abdf914805b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:28:10.053226 23092 raft_consensus.cc:482] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:28:10.053423 23092 raft_consensus.cc:504] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:10.053978 23092 leader_election.cc:288] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7743196215194f4b93a5abdf914805b2 (127.20.92.1:44555), bb810c0459974dc6a133b30479c09c91 (127.20.92.2:41717)
I20260929 10:28:10.054602 22735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbc9b98b3b2646fba9200e4c1607ea5a" candidate_uuid: "951cf90099f0458e9b85d9e47a6ed921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7743196215194f4b93a5abdf914805b2" is_pre_election: true
I20260929 10:28:10.054612 22875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbc9b98b3b2646fba9200e4c1607ea5a" candidate_uuid: "951cf90099f0458e9b85d9e47a6ed921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb810c0459974dc6a133b30479c09c91" is_pre_election: true
I20260929 10:28:10.054760 22875 raft_consensus.cc:2454] T bbc9b98b3b2646fba9200e4c1607ea5a P bb810c0459974dc6a133b30479c09c91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 951cf90099f0458e9b85d9e47a6ed921 in term 0.
I20260929 10:28:10.054760 22735 raft_consensus.cc:2454] T bbc9b98b3b2646fba9200e4c1607ea5a P 7743196215194f4b93a5abdf914805b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 951cf90099f0458e9b85d9e47a6ed921 in term 0.
I20260929 10:28:10.055127 22947 leader_election.cc:302] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7743196215194f4b93a5abdf914805b2, 951cf90099f0458e9b85d9e47a6ed921, bb810c0459974dc6a133b30479c09c91; no voters:
I20260929 10:28:10.055349 23092 raft_consensus.cc:2781] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:28:10.055442 23092 raft_consensus.cc:482] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:28:10.055490 23092 raft_consensus.cc:3037] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:10.056699 23092 raft_consensus.cc:504] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:10.057026 23092 leader_election.cc:288] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921 [CANDIDATE]: Term 1 election: Requested vote from peers 7743196215194f4b93a5abdf914805b2 (127.20.92.1:44555), bb810c0459974dc6a133b30479c09c91 (127.20.92.2:41717)
I20260929 10:28:10.057468 22875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbc9b98b3b2646fba9200e4c1607ea5a" candidate_uuid: "951cf90099f0458e9b85d9e47a6ed921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb810c0459974dc6a133b30479c09c91"
I20260929 10:28:10.057612 22875 raft_consensus.cc:3037] T bbc9b98b3b2646fba9200e4c1607ea5a P bb810c0459974dc6a133b30479c09c91 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:10.057708 22735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbc9b98b3b2646fba9200e4c1607ea5a" candidate_uuid: "951cf90099f0458e9b85d9e47a6ed921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7743196215194f4b93a5abdf914805b2"
I20260929 10:28:10.057906 22735 raft_consensus.cc:3037] T bbc9b98b3b2646fba9200e4c1607ea5a P 7743196215194f4b93a5abdf914805b2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:10.058670 22875 raft_consensus.cc:2454] T bbc9b98b3b2646fba9200e4c1607ea5a P bb810c0459974dc6a133b30479c09c91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 951cf90099f0458e9b85d9e47a6ed921 in term 1.
I20260929 10:28:10.058898 22735 raft_consensus.cc:2454] T bbc9b98b3b2646fba9200e4c1607ea5a P 7743196215194f4b93a5abdf914805b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 951cf90099f0458e9b85d9e47a6ed921 in term 1.
I20260929 10:28:10.058997 22946 leader_election.cc:302] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951cf90099f0458e9b85d9e47a6ed921, bb810c0459974dc6a133b30479c09c91; no voters:
I20260929 10:28:10.059193 23092 raft_consensus.cc:2781] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:10.059307 23092 raft_consensus.cc:686] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921 [term 1 LEADER]: Becoming Leader. State: Replica: 951cf90099f0458e9b85d9e47a6ed921, State: Running, Role: LEADER
I20260929 10:28:10.059446 23092 consensus_queue.cc:237] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:10.060177 23094 tablet_replica.cc:439] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921: TxnStatusTablet state changed. Reason: New leader 951cf90099f0458e9b85d9e47a6ed921. Latest consensus state: current_term: 1 leader_uuid: "951cf90099f0458e9b85d9e47a6ed921" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } } }
I20260929 10:28:10.060305 23094 tablet_replica.cc:442] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921: This TxnStatusTablet replica's current role is: LEADER
I20260929 10:28:10.060614 23106 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 10:28:10.060861 23106 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 10:28:10.061496 22577 catalog_manager.cc:5495] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921 reported cstate change: term changed from 0 to 1, leader changed from <none> to 951cf90099f0458e9b85d9e47a6ed921 (127.20.92.3). New cstate: current_term: 1 leader_uuid: "951cf90099f0458e9b85d9e47a6ed921" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } health_report { overall_health: HEALTHY } } }
I20260929 10:28:10.081094 23101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:10.076393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:40001 (local address 127.0.0.1:39866)
0929 10:28:10.076536 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:10.076555 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:10.076663 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:10.077013 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:10.077015 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:10.077024 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:28:10.077117 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:10.077121 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:10.078181 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:10.078184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:10.078890 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:10.078901 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:10.079572 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:10.079575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:10.079653 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:10.079709 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:28:10.080669 (+ 960us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:28:10.080733 (+ 64us) client_negotiation.cc:769] Sending connection context
0929 10:28:10.080806 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 10:28:10.080906 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 10:28:10.081094 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:10.076494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39866 (local address 127.20.92.3:40001)
0929 10:28:10.076643 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:10.076648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:10.076667 (+ 19us) server_negotiation.cc:407] Connection header received
0929 10:28:10.076809 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:10.076813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:10.076889 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:10.077007 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:28:10.077221 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:10.078051 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:10.079016 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:10.079407 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:10.079534 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:10.079913 (+ 379us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:28:10.080512 (+ 599us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:28:10.080640 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:10.080852 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 10:28:10.080925 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 10:28:10.087136 22735 raft_consensus.cc:1261] T bbc9b98b3b2646fba9200e4c1607ea5a P 7743196215194f4b93a5abdf914805b2 [term 1 FOLLOWER]: Refusing update from remote peer 951cf90099f0458e9b85d9e47a6ed921: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:10.087136 22875 raft_consensus.cc:1261] T bbc9b98b3b2646fba9200e4c1607ea5a P bb810c0459974dc6a133b30479c09c91 [term 1 FOLLOWER]: Refusing update from remote peer 951cf90099f0458e9b85d9e47a6ed921: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:10.087970 23094 consensus_queue.cc:1035] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:28:10.088488 23094 consensus_queue.cc:1035] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:28:10.093197 23090 tablet_replica.cc:439] T bbc9b98b3b2646fba9200e4c1607ea5a P 7743196215194f4b93a5abdf914805b2: TxnStatusTablet state changed. Reason: New leader 951cf90099f0458e9b85d9e47a6ed921. Latest consensus state: current_term: 1 leader_uuid: "951cf90099f0458e9b85d9e47a6ed921" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } } }
I20260929 10:28:10.093372 23090 tablet_replica.cc:442] T bbc9b98b3b2646fba9200e4c1607ea5a P 7743196215194f4b93a5abdf914805b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:28:10.093806 23091 tablet_replica.cc:439] T bbc9b98b3b2646fba9200e4c1607ea5a P bb810c0459974dc6a133b30479c09c91: TxnStatusTablet state changed. Reason: New leader 951cf90099f0458e9b85d9e47a6ed921. Latest consensus state: current_term: 1 leader_uuid: "951cf90099f0458e9b85d9e47a6ed921" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } } }
I20260929 10:28:10.093940 23091 tablet_replica.cc:442] T bbc9b98b3b2646fba9200e4c1607ea5a P bb810c0459974dc6a133b30479c09c91: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:28:10.096680 23094 tablet_replica.cc:439] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "951cf90099f0458e9b85d9e47a6ed921" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } } }
I20260929 10:28:10.096800 23094 tablet_replica.cc:442] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921: This TxnStatusTablet replica's current role is: LEADER
I20260929 10:28:10.096912 23109 mvcc.cc:205] Tried to move back new op lower bound from 7334615818588921856 to 7334615818484961280. Current Snapshot: MvccSnapshot[applied={T|T < 7334615818588921856}]
I20260929 10:28:10.097239 23090 tablet_replica.cc:439] T bbc9b98b3b2646fba9200e4c1607ea5a P 7743196215194f4b93a5abdf914805b2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "951cf90099f0458e9b85d9e47a6ed921" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } } }
I20260929 10:28:10.097365 23090 tablet_replica.cc:442] T bbc9b98b3b2646fba9200e4c1607ea5a P 7743196215194f4b93a5abdf914805b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:28:10.098642 23110 mvcc.cc:205] Tried to move back new op lower bound from 7334615818588921856 to 7334615818484961280. Current Snapshot: MvccSnapshot[applied={T|T < 7334615818588921856}]
I20260929 10:28:10.099900 23092 tablet_replica.cc:439] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "951cf90099f0458e9b85d9e47a6ed921" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } } }
I20260929 10:28:10.100020 23092 tablet_replica.cc:442] T bbc9b98b3b2646fba9200e4c1607ea5a P 951cf90099f0458e9b85d9e47a6ed921: This TxnStatusTablet replica's current role is: LEADER
I20260929 10:28:10.101202 23091 tablet_replica.cc:439] T bbc9b98b3b2646fba9200e4c1607ea5a P bb810c0459974dc6a133b30479c09c91: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "951cf90099f0458e9b85d9e47a6ed921" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } } }
I20260929 10:28:10.101321 23091 tablet_replica.cc:442] T bbc9b98b3b2646fba9200e4c1607ea5a P bb810c0459974dc6a133b30479c09c91: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:28:10.101372 23112 mvcc.cc:205] Tried to move back new op lower bound from 7334615818588921856 to 7334615818484961280. Current Snapshot: MvccSnapshot[applied={T|T < 7334615818588921856}]
I20260929 10:28:10.110127 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:10.106431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39878 (local address 127.20.92.3:40001)
0929 10:28:10.106560 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:10.106564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:10.106620 (+ 56us) server_negotiation.cc:407] Connection header received
0929 10:28:10.106760 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:10.106762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:10.106802 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:10.106871 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:28:10.107122 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:10.107637 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:10.108859 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:10.109070 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:10.109157 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:10.109500 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:28:10.109573 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:28:10.109640 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:10.109874 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 10:28:10.109956 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 10:28:10.121222 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:10.117700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:40001 (local address 127.20.92.3:49861)
0929 10:28:10.117885 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:10.117903 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:10.117988 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:10.118232 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:10.118234 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:10.118239 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:28:10.118370 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:10.118375 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:10.119349 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:10.119352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:10.119982 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:10.119991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:10.120549 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:10.120551 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:10.120634 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:10.120677 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:28:10.120981 (+ 304us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:28:10.120986 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 10:28:10.121036 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 10:28:10.121096 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":20}
I20260929 10:28:10.121251 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:10.117834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:49861 (local address 127.20.92.3:40001)
0929 10:28:10.117937 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:10.117941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:10.117956 (+ 15us) server_negotiation.cc:407] Connection header received
0929 10:28:10.118086 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:10.118089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:10.118136 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:10.118213 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:28:10.118472 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:10.119234 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:10.120094 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:10.120436 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:10.120538 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:10.120819 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:28:10.120896 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:28:10.120966 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:10.121063 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 10:28:10.121113 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":41}
I20260929 10:28:10.129148 22735 raft_consensus.cc:1261] T 38234e2f56c945f1b53788b724512df3 P 7743196215194f4b93a5abdf914805b2 [term 1 FOLLOWER]: Refusing update from remote peer 951cf90099f0458e9b85d9e47a6ed921: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:10.129245 22875 raft_consensus.cc:1261] T 38234e2f56c945f1b53788b724512df3 P bb810c0459974dc6a133b30479c09c91 [term 1 FOLLOWER]: Refusing update from remote peer 951cf90099f0458e9b85d9e47a6ed921: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:10.129534 23092 consensus_queue.cc:1035] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:28:10.129679 23116 consensus_queue.cc:1035] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:28:10.173015 22578 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:28:10.173183 22578 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:28:10.175840 22578 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac1e7544669a4d08bb861ec2c8cfd64a: Sending DeleteTablet for 3 replicas of tablet 38234e2f56c945f1b53788b724512df3
I20260929 10:28:10.176759 22715 tablet_service.cc:1514] Processing DeleteTablet for tablet 38234e2f56c945f1b53788b724512df3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:10 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:34322
I20260929 10:28:10.176843 22855 tablet_service.cc:1514] Processing DeleteTablet for tablet 38234e2f56c945f1b53788b724512df3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:10 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:58410
I20260929 10:28:10.177349 23124 tablet_replica.cc:330] T 38234e2f56c945f1b53788b724512df3 P 7743196215194f4b93a5abdf914805b2: stopping tablet replica
I20260929 10:28:10.177409 23125 tablet_replica.cc:330] T 38234e2f56c945f1b53788b724512df3 P bb810c0459974dc6a133b30479c09c91: stopping tablet replica
I20260929 10:28:10.177765 23125 raft_consensus.cc:2229] T 38234e2f56c945f1b53788b724512df3 P bb810c0459974dc6a133b30479c09c91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:28:10.177805 23124 raft_consensus.cc:2229] T 38234e2f56c945f1b53788b724512df3 P 7743196215194f4b93a5abdf914805b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:28:10.178001 23125 raft_consensus.cc:2258] T 38234e2f56c945f1b53788b724512df3 P bb810c0459974dc6a133b30479c09c91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:10.178211 23124 raft_consensus.cc:2258] T 38234e2f56c945f1b53788b724512df3 P 7743196215194f4b93a5abdf914805b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:10.178182 22578 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:28:10.178732 22578 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:28:10.179646 22995 tablet_service.cc:1514] Processing DeleteTablet for tablet 38234e2f56c945f1b53788b724512df3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:10 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:39858
I20260929 10:28:10.179868 23124 ts_tablet_manager.cc:1905] T 38234e2f56c945f1b53788b724512df3 P 7743196215194f4b93a5abdf914805b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:10.181648 23125 ts_tablet_manager.cc:1905] T 38234e2f56c945f1b53788b724512df3 P bb810c0459974dc6a133b30479c09c91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:10.183604 23124 ts_tablet_manager.cc:1918] T 38234e2f56c945f1b53788b724512df3 P 7743196215194f4b93a5abdf914805b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:28:10.183717 23124 log.cc:1196] T 38234e2f56c945f1b53788b724512df3 P 7743196215194f4b93a5abdf914805b2: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-0/wal/wals/38234e2f56c945f1b53788b724512df3
I20260929 10:28:10.184125 23124 ts_tablet_manager.cc:1939] T 38234e2f56c945f1b53788b724512df3 P 7743196215194f4b93a5abdf914805b2: Deleting consensus metadata
I20260929 10:28:10.184732 22715 tablet_service.cc:1467] Processing CreateTablet for tablet afe526d50a6f4b7681c758b774e83552 (DEFAULT_TABLE table=test-table [id=0fa29ebc66844c2ebda688a65a6a809d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:10.185022 22715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afe526d50a6f4b7681c758b774e83552. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:10.185833 23126 tablet_replica.cc:330] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921: stopping tablet replica
I20260929 10:28:10.186101 22563 catalog_manager.cc:4841] TS 7743196215194f4b93a5abdf914805b2 (127.20.92.1:44555): tablet 38234e2f56c945f1b53788b724512df3 (table test-table [id=f5e11d7eb89749d8a6f288fa16771d3b]) successfully deleted
I20260929 10:28:10.186199 23126 raft_consensus.cc:2229] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:28:10.186589 23126 raft_consensus.cc:2258] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:10.187029 23125 ts_tablet_manager.cc:1918] T 38234e2f56c945f1b53788b724512df3 P bb810c0459974dc6a133b30479c09c91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:28:10.187186 23125 log.cc:1196] T 38234e2f56c945f1b53788b724512df3 P bb810c0459974dc6a133b30479c09c91: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-1/wal/wals/38234e2f56c945f1b53788b724512df3
I20260929 10:28:10.187610 23125 ts_tablet_manager.cc:1939] T 38234e2f56c945f1b53788b724512df3 P bb810c0459974dc6a133b30479c09c91: Deleting consensus metadata
I20260929 10:28:10.188578 23126 ts_tablet_manager.cc:1905] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:10.189101 22562 catalog_manager.cc:4841] TS bb810c0459974dc6a133b30479c09c91 (127.20.92.2:41717): tablet 38234e2f56c945f1b53788b724512df3 (table test-table [id=f5e11d7eb89749d8a6f288fa16771d3b]) successfully deleted
I20260929 10:28:10.189389 22855 tablet_service.cc:1467] Processing CreateTablet for tablet afe526d50a6f4b7681c758b774e83552 (DEFAULT_TABLE table=test-table [id=0fa29ebc66844c2ebda688a65a6a809d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:10.189649 22855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afe526d50a6f4b7681c758b774e83552. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:10.190450 22995 tablet_service.cc:1467] Processing CreateTablet for tablet afe526d50a6f4b7681c758b774e83552 (DEFAULT_TABLE table=test-table [id=0fa29ebc66844c2ebda688a65a6a809d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:10.190790 22995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afe526d50a6f4b7681c758b774e83552. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:10.191690 23126 ts_tablet_manager.cc:1918] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:28:10.191802 23126 log.cc:1196] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-2/wal/wals/38234e2f56c945f1b53788b724512df3
I20260929 10:28:10.192068 23084 tablet_bootstrap.cc:492] T afe526d50a6f4b7681c758b774e83552 P 7743196215194f4b93a5abdf914805b2: Bootstrap starting.
I20260929 10:28:10.192143 23126 ts_tablet_manager.cc:1939] T 38234e2f56c945f1b53788b724512df3 P 951cf90099f0458e9b85d9e47a6ed921: Deleting consensus metadata
I20260929 10:28:10.193216 23084 tablet_bootstrap.cc:654] T afe526d50a6f4b7681c758b774e83552 P 7743196215194f4b93a5abdf914805b2: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:10.193532 23086 tablet_bootstrap.cc:492] T afe526d50a6f4b7681c758b774e83552 P 951cf90099f0458e9b85d9e47a6ed921: Bootstrap starting.
I20260929 10:28:10.194473 22564 catalog_manager.cc:4841] TS 951cf90099f0458e9b85d9e47a6ed921 (127.20.92.3:40001): tablet 38234e2f56c945f1b53788b724512df3 (table test-table [id=f5e11d7eb89749d8a6f288fa16771d3b]) successfully deleted
I20260929 10:28:10.194537 23086 tablet_bootstrap.cc:654] T afe526d50a6f4b7681c758b774e83552 P 951cf90099f0458e9b85d9e47a6ed921: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:10.195317 23084 tablet_bootstrap.cc:492] T afe526d50a6f4b7681c758b774e83552 P 7743196215194f4b93a5abdf914805b2: No bootstrap required, opened a new log
I20260929 10:28:10.195456 23084 ts_tablet_manager.cc:1397] T afe526d50a6f4b7681c758b774e83552 P 7743196215194f4b93a5abdf914805b2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 10:28:10.195806 23086 tablet_bootstrap.cc:492] T afe526d50a6f4b7681c758b774e83552 P 951cf90099f0458e9b85d9e47a6ed921: No bootstrap required, opened a new log
I20260929 10:28:10.195886 23086 ts_tablet_manager.cc:1397] T afe526d50a6f4b7681c758b774e83552 P 951cf90099f0458e9b85d9e47a6ed921: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:28:10.196020 23084 raft_consensus.cc:348] T afe526d50a6f4b7681c758b774e83552 P 7743196215194f4b93a5abdf914805b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:10.196148 23084 raft_consensus.cc:374] T afe526d50a6f4b7681c758b774e83552 P 7743196215194f4b93a5abdf914805b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:10.196210 23084 raft_consensus.cc:729] T afe526d50a6f4b7681c758b774e83552 P 7743196215194f4b93a5abdf914805b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7743196215194f4b93a5abdf914805b2, State: Initialized, Role: FOLLOWER
I20260929 10:28:10.196350 23086 raft_consensus.cc:348] T afe526d50a6f4b7681c758b774e83552 P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:10.196475 23086 raft_consensus.cc:374] T afe526d50a6f4b7681c758b774e83552 P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:10.196429 23084 consensus_queue.cc:260] T afe526d50a6f4b7681c758b774e83552 P 7743196215194f4b93a5abdf914805b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:10.196506 23086 raft_consensus.cc:729] T afe526d50a6f4b7681c758b774e83552 P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951cf90099f0458e9b85d9e47a6ed921, State: Initialized, Role: FOLLOWER
I20260929 10:28:10.196619 23086 consensus_queue.cc:260] T afe526d50a6f4b7681c758b774e83552 P 951cf90099f0458e9b85d9e47a6ed921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:10.196913 23086 ts_tablet_manager.cc:1428] T afe526d50a6f4b7681c758b774e83552 P 951cf90099f0458e9b85d9e47a6ed921: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 10:28:10.197381 23085 tablet_bootstrap.cc:492] T afe526d50a6f4b7681c758b774e83552 P bb810c0459974dc6a133b30479c09c91: Bootstrap starting.
I20260929 10:28:10.198462 23085 tablet_bootstrap.cc:654] T afe526d50a6f4b7681c758b774e83552 P bb810c0459974dc6a133b30479c09c91: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:10.199806 23085 tablet_bootstrap.cc:492] T afe526d50a6f4b7681c758b774e83552 P bb810c0459974dc6a133b30479c09c91: No bootstrap required, opened a new log
I20260929 10:28:10.199915 23085 ts_tablet_manager.cc:1397] T afe526d50a6f4b7681c758b774e83552 P bb810c0459974dc6a133b30479c09c91: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 10:28:10.200299 23085 raft_consensus.cc:348] T afe526d50a6f4b7681c758b774e83552 P bb810c0459974dc6a133b30479c09c91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:10.200415 23085 raft_consensus.cc:374] T afe526d50a6f4b7681c758b774e83552 P bb810c0459974dc6a133b30479c09c91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:10.200459 23085 raft_consensus.cc:729] T afe526d50a6f4b7681c758b774e83552 P bb810c0459974dc6a133b30479c09c91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb810c0459974dc6a133b30479c09c91, State: Initialized, Role: FOLLOWER
I20260929 10:28:10.200587 23085 consensus_queue.cc:260] T afe526d50a6f4b7681c758b774e83552 P bb810c0459974dc6a133b30479c09c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:10.201509 23085 ts_tablet_manager.cc:1428] T afe526d50a6f4b7681c758b774e83552 P bb810c0459974dc6a133b30479c09c91: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 10:28:10.203035 23084 ts_tablet_manager.cc:1428] T afe526d50a6f4b7681c758b774e83552 P 7743196215194f4b93a5abdf914805b2: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20260929 10:28:10.327699 23091 raft_consensus.cc:482] T afe526d50a6f4b7681c758b774e83552 P bb810c0459974dc6a133b30479c09c91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:28:10.327853 23091 raft_consensus.cc:504] T afe526d50a6f4b7681c758b774e83552 P bb810c0459974dc6a133b30479c09c91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:10.329228 23091 leader_election.cc:288] T afe526d50a6f4b7681c758b774e83552 P bb810c0459974dc6a133b30479c09c91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7743196215194f4b93a5abdf914805b2 (127.20.92.1:44555), 951cf90099f0458e9b85d9e47a6ed921 (127.20.92.3:40001)
I20260929 10:28:10.334391 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:10.329788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:57113 (local address 127.20.92.1:44555)
0929 10:28:10.329953 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:10.329959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:10.330065 (+ 106us) server_negotiation.cc:407] Connection header received
0929 10:28:10.330335 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:10.330340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:10.330397 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:10.330442 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:10.330889 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:10.331771 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:10.333074 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:10.333705 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:10.333854 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:10.333927 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:10.334169 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 10:28:10.334249 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 10:28:10.334419 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:10.329493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:44555 (local address 127.20.92.2:57113)
0929 10:28:10.329935 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:10.329952 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:10.330164 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:10.330612 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:10.330615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:10.330653 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:10.330747 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:10.330754 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:10.331905 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:10.331908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:10.332934 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:10.332945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:10.333856 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:10.333863 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:10.334014 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:10.334047 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 10:28:10.334148 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 10:28:10.334230 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"spinlock_wait_cycles":3200,"thread_start_us":83,"threads_started":1}
I20260929 10:28:10.335157 22735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afe526d50a6f4b7681c758b774e83552" candidate_uuid: "bb810c0459974dc6a133b30479c09c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7743196215194f4b93a5abdf914805b2" is_pre_election: true
I20260929 10:28:10.335337 22735 raft_consensus.cc:2454] T afe526d50a6f4b7681c758b774e83552 P 7743196215194f4b93a5abdf914805b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb810c0459974dc6a133b30479c09c91 in term 0.
I20260929 10:28:10.335391 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:10.329939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:32771 (local address 127.20.92.3:40001)
0929 10:28:10.330070 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:10.330075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:10.330231 (+ 156us) server_negotiation.cc:407] Connection header received
0929 10:28:10.330342 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:10.330346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:10.330404 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:10.330510 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:10.331937 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:10.332624 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:10.334003 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:10.334583 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:10.334718 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:10.334785 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:10.335169 (+ 384us) server_negotiation.cc:299] Negotiation successful
0929 10:28:10.335252 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 10:28:10.335503 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:10.329505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:40001 (local address 127.20.92.2:32771)
0929 10:28:10.330125 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:10.330139 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:10.330234 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:10.330557 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:10.330564 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:10.330588 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:10.330747 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:10.330754 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:10.332793 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:10.332797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:10.333718 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:10.333728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:10.334891 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:10.334895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:10.335000 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:10.335034 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 10:28:10.335148 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 10:28:10.335232 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"spinlock_wait_cycles":23296,"thread_start_us":113,"threads_started":1}
I20260929 10:28:10.335789 22807 leader_election.cc:302] T afe526d50a6f4b7681c758b774e83552 P bb810c0459974dc6a133b30479c09c91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7743196215194f4b93a5abdf914805b2, bb810c0459974dc6a133b30479c09c91; no voters:
I20260929 10:28:10.336112 23091 raft_consensus.cc:2781] T afe526d50a6f4b7681c758b774e83552 P bb810c0459974dc6a133b30479c09c91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:28:10.336179 23091 raft_consensus.cc:482] T afe526d50a6f4b7681c758b774e83552 P bb810c0459974dc6a133b30479c09c91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:28:10.336223 23091 raft_consensus.cc:3037] T afe526d50a6f4b7681c758b774e83552 P bb810c0459974dc6a133b30479c09c91 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:10.336122 23015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afe526d50a6f4b7681c758b774e83552" candidate_uuid: "bb810c0459974dc6a133b30479c09c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951cf90099f0458e9b85d9e47a6ed921" is_pre_election: true
I20260929 10:28:10.336431 23015 raft_consensus.cc:2454] T afe526d50a6f4b7681c758b774e83552 P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb810c0459974dc6a133b30479c09c91 in term 0.
I20260929 10:28:10.337572 23091 raft_consensus.cc:504] T afe526d50a6f4b7681c758b774e83552 P bb810c0459974dc6a133b30479c09c91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:10.338042 23091 leader_election.cc:288] T afe526d50a6f4b7681c758b774e83552 P bb810c0459974dc6a133b30479c09c91 [CANDIDATE]: Term 1 election: Requested vote from peers 7743196215194f4b93a5abdf914805b2 (127.20.92.1:44555), 951cf90099f0458e9b85d9e47a6ed921 (127.20.92.3:40001)
I20260929 10:28:10.338454 23015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afe526d50a6f4b7681c758b774e83552" candidate_uuid: "bb810c0459974dc6a133b30479c09c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951cf90099f0458e9b85d9e47a6ed921"
I20260929 10:28:10.338534 22735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afe526d50a6f4b7681c758b774e83552" candidate_uuid: "bb810c0459974dc6a133b30479c09c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7743196215194f4b93a5abdf914805b2"
I20260929 10:28:10.338599 23015 raft_consensus.cc:3037] T afe526d50a6f4b7681c758b774e83552 P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:10.338675 22735 raft_consensus.cc:3037] T afe526d50a6f4b7681c758b774e83552 P 7743196215194f4b93a5abdf914805b2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:10.339540 23015 raft_consensus.cc:2454] T afe526d50a6f4b7681c758b774e83552 P 951cf90099f0458e9b85d9e47a6ed921 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb810c0459974dc6a133b30479c09c91 in term 1.
I20260929 10:28:10.339804 22735 raft_consensus.cc:2454] T afe526d50a6f4b7681c758b774e83552 P 7743196215194f4b93a5abdf914805b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb810c0459974dc6a133b30479c09c91 in term 1.
I20260929 10:28:10.340291 22808 leader_election.cc:302] T afe526d50a6f4b7681c758b774e83552 P bb810c0459974dc6a133b30479c09c91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951cf90099f0458e9b85d9e47a6ed921, bb810c0459974dc6a133b30479c09c91; no voters:
I20260929 10:28:10.340498 23091 raft_consensus.cc:2781] T afe526d50a6f4b7681c758b774e83552 P bb810c0459974dc6a133b30479c09c91 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:10.340731 23091 raft_consensus.cc:686] T afe526d50a6f4b7681c758b774e83552 P bb810c0459974dc6a133b30479c09c91 [term 1 LEADER]: Becoming Leader. State: Replica: bb810c0459974dc6a133b30479c09c91, State: Running, Role: LEADER
I20260929 10:28:10.341002 23091 consensus_queue.cc:237] T afe526d50a6f4b7681c758b774e83552 P bb810c0459974dc6a133b30479c09c91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:10.343879 22578 catalog_manager.cc:5495] T afe526d50a6f4b7681c758b774e83552 P bb810c0459974dc6a133b30479c09c91 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb810c0459974dc6a133b30479c09c91 (127.20.92.2). New cstate: current_term: 1 leader_uuid: "bb810c0459974dc6a133b30479c09c91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } health_report { overall_health: UNKNOWN } } }
I20260929 10:28:10.365926 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:10.361355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58414 (local address 127.20.92.2:41717)
0929 10:28:10.361494 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:10.361499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:10.361513 (+ 14us) server_negotiation.cc:407] Connection header received
0929 10:28:10.361708 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:10.361711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:10.361846 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:10.361959 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:28:10.362267 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:10.362871 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:10.364051 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:10.364290 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:10.364378 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:10.364722 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:28:10.365225 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:28:10.365326 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:10.365565 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 10:28:10.365633 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 10:28:10.380031 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:10.375516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:40001 (local address 127.20.92.2:37795)
0929 10:28:10.375865 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:10.375882 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:10.376016 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:10.376373 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:10.376377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:10.376386 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:28:10.376536 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:10.376544 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:10.377424 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:10.377428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:10.378440 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:10.378452 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:10.378992 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:10.378997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:10.379106 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:10.379174 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:28:10.379625 (+ 451us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:28:10.379634 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 10:28:10.379725 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 10:28:10.379821 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":118,"threads_started":1}
I20260929 10:28:10.380052 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:10.375662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:37795 (local address 127.20.92.3:40001)
0929 10:28:10.375869 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:10.375873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:10.375965 (+ 92us) server_negotiation.cc:407] Connection header received
0929 10:28:10.376181 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:10.376184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:10.376236 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:10.376315 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:28:10.376752 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:10.377285 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:10.378590 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:10.378852 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:10.378952 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:10.379420 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:28:10.379506 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:28:10.379586 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:10.379852 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 10:28:10.379916 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260929 10:28:10.387286 23015 raft_consensus.cc:1261] T afe526d50a6f4b7681c758b774e83552 P 951cf90099f0458e9b85d9e47a6ed921 [term 1 FOLLOWER]: Refusing update from remote peer bb810c0459974dc6a133b30479c09c91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:10.387771 22735 raft_consensus.cc:1261] T afe526d50a6f4b7681c758b774e83552 P 7743196215194f4b93a5abdf914805b2 [term 1 FOLLOWER]: Refusing update from remote peer bb810c0459974dc6a133b30479c09c91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:10.387957 23132 consensus_queue.cc:1035] T afe526d50a6f4b7681c758b774e83552 P bb810c0459974dc6a133b30479c09c91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:28:10.388549 23132 consensus_queue.cc:1035] T afe526d50a6f4b7681c758b774e83552 P bb810c0459974dc6a133b30479c09c91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:28:10.407652 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:10.403837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:41717 (local address 127.20.92.3:50301)
0929 10:28:10.403995 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:10.404009 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:10.404081 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:10.404343 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:10.404346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:10.404353 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:28:10.404466 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:10.404472 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:10.405310 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:10.405314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:10.406136 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:10.406149 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:10.406753 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:10.406757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:10.406905 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:10.406938 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:28:10.407310 (+ 372us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:28:10.407337 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 10:28:10.407410 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 10:28:10.407507 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 10:28:10.408020 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:10.403922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:50301 (local address 127.20.92.2:41717)
0929 10:28:10.404028 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:10.404033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:10.404047 (+ 14us) server_negotiation.cc:407] Connection header received
0929 10:28:10.404176 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:10.404179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:10.404231 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:10.404333 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:28:10.404576 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:10.405185 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:10.406275 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:10.406636 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:10.406757 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:10.407111 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:28:10.407213 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:28:10.407301 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:10.407873 (+ 572us) server_negotiation.cc:299] Negotiation successful
0929 10:28:10.407933 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260929 10:28:10.408663 23112 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 10:28:11.973459 22578 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:28:11.973640 22578 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:28:11.976603 22578 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac1e7544669a4d08bb861ec2c8cfd64a: Sending DeleteTablet for 3 replicas of tablet afe526d50a6f4b7681c758b774e83552
I20260929 10:28:11.977270 22715 tablet_service.cc:1514] Processing DeleteTablet for tablet afe526d50a6f4b7681c758b774e83552 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:11 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:34322
I20260929 10:28:11.977542 22994 tablet_service.cc:1514] Processing DeleteTablet for tablet afe526d50a6f4b7681c758b774e83552 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:11 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:39858
I20260929 10:28:11.977679 23163 tablet_replica.cc:330] T afe526d50a6f4b7681c758b774e83552 P 7743196215194f4b93a5abdf914805b2: stopping tablet replica
I20260929 10:28:11.977869 23163 raft_consensus.cc:2229] T afe526d50a6f4b7681c758b774e83552 P 7743196215194f4b93a5abdf914805b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:28:11.978020 23163 raft_consensus.cc:2258] T afe526d50a6f4b7681c758b774e83552 P 7743196215194f4b93a5abdf914805b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:11.978021 23164 tablet_replica.cc:330] T afe526d50a6f4b7681c758b774e83552 P 951cf90099f0458e9b85d9e47a6ed921: stopping tablet replica
I20260929 10:28:11.978163 23164 raft_consensus.cc:2229] T afe526d50a6f4b7681c758b774e83552 P 951cf90099f0458e9b85d9e47a6ed921 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:28:11.978296 23164 raft_consensus.cc:2258] T afe526d50a6f4b7681c758b774e83552 P 951cf90099f0458e9b85d9e47a6ed921 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:11.978430 22855 tablet_service.cc:1514] Processing DeleteTablet for tablet afe526d50a6f4b7681c758b774e83552 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:11 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:58410
I20260929 10:28:11.978799 23165 tablet_replica.cc:330] T afe526d50a6f4b7681c758b774e83552 P bb810c0459974dc6a133b30479c09c91: stopping tablet replica
I20260929 10:28:11.978921 23165 raft_consensus.cc:2229] T afe526d50a6f4b7681c758b774e83552 P bb810c0459974dc6a133b30479c09c91 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:28:11.979142 23164 ts_tablet_manager.cc:1905] T afe526d50a6f4b7681c758b774e83552 P 951cf90099f0458e9b85d9e47a6ed921: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:11.979269 23165 raft_consensus.cc:2258] T afe526d50a6f4b7681c758b774e83552 P bb810c0459974dc6a133b30479c09c91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:11.979974 23165 ts_tablet_manager.cc:1905] T afe526d50a6f4b7681c758b774e83552 P bb810c0459974dc6a133b30479c09c91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:11.982597 23164 ts_tablet_manager.cc:1918] T afe526d50a6f4b7681c758b774e83552 P 951cf90099f0458e9b85d9e47a6ed921: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 10:28:11.982704 23164 log.cc:1196] T afe526d50a6f4b7681c758b774e83552 P 951cf90099f0458e9b85d9e47a6ed921: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-2/wal/wals/afe526d50a6f4b7681c758b774e83552
I20260929 10:28:11.982985 23164 ts_tablet_manager.cc:1939] T afe526d50a6f4b7681c758b774e83552 P 951cf90099f0458e9b85d9e47a6ed921: Deleting consensus metadata
I20260929 10:28:11.982963 23163 ts_tablet_manager.cc:1905] T afe526d50a6f4b7681c758b774e83552 P 7743196215194f4b93a5abdf914805b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:11.983145 23165 ts_tablet_manager.cc:1918] T afe526d50a6f4b7681c758b774e83552 P bb810c0459974dc6a133b30479c09c91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 10:28:11.983225 23165 log.cc:1196] T afe526d50a6f4b7681c758b774e83552 P bb810c0459974dc6a133b30479c09c91: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-1/wal/wals/afe526d50a6f4b7681c758b774e83552
I20260929 10:28:11.983577 23165 ts_tablet_manager.cc:1939] T afe526d50a6f4b7681c758b774e83552 P bb810c0459974dc6a133b30479c09c91: Deleting consensus metadata
I20260929 10:28:11.983845 22564 catalog_manager.cc:4841] TS 951cf90099f0458e9b85d9e47a6ed921 (127.20.92.3:40001): tablet afe526d50a6f4b7681c758b774e83552 (table test-table [id=0fa29ebc66844c2ebda688a65a6a809d]) successfully deleted
I20260929 10:28:11.984344 22562 catalog_manager.cc:4841] TS bb810c0459974dc6a133b30479c09c91 (127.20.92.2:41717): tablet afe526d50a6f4b7681c758b774e83552 (table test-table [id=0fa29ebc66844c2ebda688a65a6a809d]) successfully deleted
I20260929 10:28:11.985481 23163 ts_tablet_manager.cc:1918] T afe526d50a6f4b7681c758b774e83552 P 7743196215194f4b93a5abdf914805b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 10:28:11.985579 23163 log.cc:1196] T afe526d50a6f4b7681c758b774e83552 P 7743196215194f4b93a5abdf914805b2: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-0/wal/wals/afe526d50a6f4b7681c758b774e83552
I20260929 10:28:11.985935 23163 ts_tablet_manager.cc:1939] T afe526d50a6f4b7681c758b774e83552 P 7743196215194f4b93a5abdf914805b2: Deleting consensus metadata
I20260929 10:28:11.986632 22563 catalog_manager.cc:4841] TS 7743196215194f4b93a5abdf914805b2 (127.20.92.1:44555): tablet afe526d50a6f4b7681c758b774e83552 (table test-table [id=0fa29ebc66844c2ebda688a65a6a809d]) successfully deleted
Sep 29 10:28:11 dist-test-slave-9cj5 krb5kdc[22533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677691, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 10:28:12 dist-test-slave-9cj5 krb5kdc[22533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677691, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:12.014411 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:12.001541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45712 (local address 127.20.92.62:38219)
0929 10:28:12.001965 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:12.001972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:12.001987 (+ 15us) server_negotiation.cc:407] Connection header received
0929 10:28:12.002032 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:12.002035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:12.002077 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:12.002175 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:12.003117 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:12.003670 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:12.004727 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:12.005007 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:12.005141 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:12.009058 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:12.009064 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:12.009066 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:12.009100 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:12.012105 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:12.012857 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:12.012860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:12.012862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:12.012957 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:12.013298 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:12.013301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:12.013302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:12.013597 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:12.013716 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:12.014095 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 10:28:12.014234 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":269,"threads_started":1}
I20260929 10:28:12.018164 22578 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:28:12.018739 22578 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:28:12.024194 22715 tablet_service.cc:1467] Processing CreateTablet for tablet b7b4e5cab76547179f018b987b210b09 (DEFAULT_TABLE table=test-table [id=de666e28996b491f877b89b05e2600ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:12.024487 22994 tablet_service.cc:1467] Processing CreateTablet for tablet b7b4e5cab76547179f018b987b210b09 (DEFAULT_TABLE table=test-table [id=de666e28996b491f877b89b05e2600ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:12.024654 22715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7b4e5cab76547179f018b987b210b09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:12.024753 22994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7b4e5cab76547179f018b987b210b09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:12.026063 22855 tablet_service.cc:1467] Processing CreateTablet for tablet b7b4e5cab76547179f018b987b210b09 (DEFAULT_TABLE table=test-table [id=de666e28996b491f877b89b05e2600ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:12.026417 22855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7b4e5cab76547179f018b987b210b09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:12.027873 23174 tablet_bootstrap.cc:492] T b7b4e5cab76547179f018b987b210b09 P 7743196215194f4b93a5abdf914805b2: Bootstrap starting.
I20260929 10:28:12.028088 23175 tablet_bootstrap.cc:492] T b7b4e5cab76547179f018b987b210b09 P 951cf90099f0458e9b85d9e47a6ed921: Bootstrap starting.
I20260929 10:28:12.028927 23175 tablet_bootstrap.cc:654] T b7b4e5cab76547179f018b987b210b09 P 951cf90099f0458e9b85d9e47a6ed921: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:12.029237 23174 tablet_bootstrap.cc:654] T b7b4e5cab76547179f018b987b210b09 P 7743196215194f4b93a5abdf914805b2: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:12.029564 23176 tablet_bootstrap.cc:492] T b7b4e5cab76547179f018b987b210b09 P bb810c0459974dc6a133b30479c09c91: Bootstrap starting.
I20260929 10:28:12.030567 23174 tablet_bootstrap.cc:492] T b7b4e5cab76547179f018b987b210b09 P 7743196215194f4b93a5abdf914805b2: No bootstrap required, opened a new log
I20260929 10:28:12.030575 23175 tablet_bootstrap.cc:492] T b7b4e5cab76547179f018b987b210b09 P 951cf90099f0458e9b85d9e47a6ed921: No bootstrap required, opened a new log
I20260929 10:28:12.030673 23175 ts_tablet_manager.cc:1397] T b7b4e5cab76547179f018b987b210b09 P 951cf90099f0458e9b85d9e47a6ed921: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 10:28:12.030682 23176 tablet_bootstrap.cc:654] T b7b4e5cab76547179f018b987b210b09 P bb810c0459974dc6a133b30479c09c91: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:12.030699 23174 ts_tablet_manager.cc:1397] T b7b4e5cab76547179f018b987b210b09 P 7743196215194f4b93a5abdf914805b2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 10:28:12.031131 23174 raft_consensus.cc:348] T b7b4e5cab76547179f018b987b210b09 P 7743196215194f4b93a5abdf914805b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:12.031263 23174 raft_consensus.cc:374] T b7b4e5cab76547179f018b987b210b09 P 7743196215194f4b93a5abdf914805b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:12.031309 23174 raft_consensus.cc:729] T b7b4e5cab76547179f018b987b210b09 P 7743196215194f4b93a5abdf914805b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7743196215194f4b93a5abdf914805b2, State: Initialized, Role: FOLLOWER
I20260929 10:28:12.031442 23174 consensus_queue.cc:260] T b7b4e5cab76547179f018b987b210b09 P 7743196215194f4b93a5abdf914805b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:12.031627 23175 raft_consensus.cc:348] T b7b4e5cab76547179f018b987b210b09 P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:12.031891 23174 ts_tablet_manager.cc:1428] T b7b4e5cab76547179f018b987b210b09 P 7743196215194f4b93a5abdf914805b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 10:28:12.031926 23175 raft_consensus.cc:374] T b7b4e5cab76547179f018b987b210b09 P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:12.031975 23175 raft_consensus.cc:729] T b7b4e5cab76547179f018b987b210b09 P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951cf90099f0458e9b85d9e47a6ed921, State: Initialized, Role: FOLLOWER
I20260929 10:28:12.032318 23175 consensus_queue.cc:260] T b7b4e5cab76547179f018b987b210b09 P 951cf90099f0458e9b85d9e47a6ed921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:12.033217 23176 tablet_bootstrap.cc:492] T b7b4e5cab76547179f018b987b210b09 P bb810c0459974dc6a133b30479c09c91: No bootstrap required, opened a new log
I20260929 10:28:12.033346 23176 ts_tablet_manager.cc:1397] T b7b4e5cab76547179f018b987b210b09 P bb810c0459974dc6a133b30479c09c91: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 10:28:12.033903 23176 raft_consensus.cc:348] T b7b4e5cab76547179f018b987b210b09 P bb810c0459974dc6a133b30479c09c91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:12.034045 23176 raft_consensus.cc:374] T b7b4e5cab76547179f018b987b210b09 P bb810c0459974dc6a133b30479c09c91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:12.034094 23176 raft_consensus.cc:729] T b7b4e5cab76547179f018b987b210b09 P bb810c0459974dc6a133b30479c09c91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb810c0459974dc6a133b30479c09c91, State: Initialized, Role: FOLLOWER
I20260929 10:28:12.033694 23175 ts_tablet_manager.cc:1428] T b7b4e5cab76547179f018b987b210b09 P 951cf90099f0458e9b85d9e47a6ed921: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 10:28:12.034230 23176 consensus_queue.cc:260] T b7b4e5cab76547179f018b987b210b09 P bb810c0459974dc6a133b30479c09c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:12.034576 23176 ts_tablet_manager.cc:1428] T b7b4e5cab76547179f018b987b210b09 P bb810c0459974dc6a133b30479c09c91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:28:12.284700 23180 raft_consensus.cc:482] T b7b4e5cab76547179f018b987b210b09 P 7743196215194f4b93a5abdf914805b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:28:12.284871 23180 raft_consensus.cc:504] T b7b4e5cab76547179f018b987b210b09 P 7743196215194f4b93a5abdf914805b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:12.285357 23158 raft_consensus.cc:482] T b7b4e5cab76547179f018b987b210b09 P bb810c0459974dc6a133b30479c09c91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:28:12.285528 23158 raft_consensus.cc:504] T b7b4e5cab76547179f018b987b210b09 P bb810c0459974dc6a133b30479c09c91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:12.286125 23158 leader_election.cc:288] T b7b4e5cab76547179f018b987b210b09 P bb810c0459974dc6a133b30479c09c91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7743196215194f4b93a5abdf914805b2 (127.20.92.1:44555), 951cf90099f0458e9b85d9e47a6ed921 (127.20.92.3:40001)
I20260929 10:28:12.286476 23180 leader_election.cc:288] T b7b4e5cab76547179f018b987b210b09 P 7743196215194f4b93a5abdf914805b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb810c0459974dc6a133b30479c09c91 (127.20.92.2:41717), 951cf90099f0458e9b85d9e47a6ed921 (127.20.92.3:40001)
I20260929 10:28:12.286603 22735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7b4e5cab76547179f018b987b210b09" candidate_uuid: "bb810c0459974dc6a133b30479c09c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7743196215194f4b93a5abdf914805b2" is_pre_election: true
I20260929 10:28:12.286607 23015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7b4e5cab76547179f018b987b210b09" candidate_uuid: "bb810c0459974dc6a133b30479c09c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951cf90099f0458e9b85d9e47a6ed921" is_pre_election: true
I20260929 10:28:12.286736 22735 raft_consensus.cc:2454] T b7b4e5cab76547179f018b987b210b09 P 7743196215194f4b93a5abdf914805b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb810c0459974dc6a133b30479c09c91 in term 0.
I20260929 10:28:12.286792 23015 raft_consensus.cc:2454] T b7b4e5cab76547179f018b987b210b09 P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb810c0459974dc6a133b30479c09c91 in term 0.
I20260929 10:28:12.287209 22808 leader_election.cc:302] T b7b4e5cab76547179f018b987b210b09 P bb810c0459974dc6a133b30479c09c91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951cf90099f0458e9b85d9e47a6ed921, bb810c0459974dc6a133b30479c09c91; no voters:
I20260929 10:28:12.287505 23158 raft_consensus.cc:2781] T b7b4e5cab76547179f018b987b210b09 P bb810c0459974dc6a133b30479c09c91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:28:12.287568 23158 raft_consensus.cc:482] T b7b4e5cab76547179f018b987b210b09 P bb810c0459974dc6a133b30479c09c91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:28:12.287609 23158 raft_consensus.cc:3037] T b7b4e5cab76547179f018b987b210b09 P bb810c0459974dc6a133b30479c09c91 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:12.289011 23158 raft_consensus.cc:504] T b7b4e5cab76547179f018b987b210b09 P bb810c0459974dc6a133b30479c09c91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:12.289395 23158 leader_election.cc:288] T b7b4e5cab76547179f018b987b210b09 P bb810c0459974dc6a133b30479c09c91 [CANDIDATE]: Term 1 election: Requested vote from peers 7743196215194f4b93a5abdf914805b2 (127.20.92.1:44555), 951cf90099f0458e9b85d9e47a6ed921 (127.20.92.3:40001)
I20260929 10:28:12.289811 22735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7b4e5cab76547179f018b987b210b09" candidate_uuid: "bb810c0459974dc6a133b30479c09c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7743196215194f4b93a5abdf914805b2"
I20260929 10:28:12.289948 22735 raft_consensus.cc:3037] T b7b4e5cab76547179f018b987b210b09 P 7743196215194f4b93a5abdf914805b2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:12.290019 23015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7b4e5cab76547179f018b987b210b09" candidate_uuid: "bb810c0459974dc6a133b30479c09c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951cf90099f0458e9b85d9e47a6ed921"
I20260929 10:28:12.290141 23015 raft_consensus.cc:3037] T b7b4e5cab76547179f018b987b210b09 P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:12.291049 22735 raft_consensus.cc:2454] T b7b4e5cab76547179f018b987b210b09 P 7743196215194f4b93a5abdf914805b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb810c0459974dc6a133b30479c09c91 in term 1.
I20260929 10:28:12.291337 22807 leader_election.cc:302] T b7b4e5cab76547179f018b987b210b09 P bb810c0459974dc6a133b30479c09c91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7743196215194f4b93a5abdf914805b2, bb810c0459974dc6a133b30479c09c91; no voters:
I20260929 10:28:12.291392 23015 raft_consensus.cc:2454] T b7b4e5cab76547179f018b987b210b09 P 951cf90099f0458e9b85d9e47a6ed921 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb810c0459974dc6a133b30479c09c91 in term 1.
I20260929 10:28:12.291599 23158 raft_consensus.cc:2781] T b7b4e5cab76547179f018b987b210b09 P bb810c0459974dc6a133b30479c09c91 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:12.291808 23158 raft_consensus.cc:686] T b7b4e5cab76547179f018b987b210b09 P bb810c0459974dc6a133b30479c09c91 [term 1 LEADER]: Becoming Leader. State: Replica: bb810c0459974dc6a133b30479c09c91, State: Running, Role: LEADER
I20260929 10:28:12.291963 23158 consensus_queue.cc:237] T b7b4e5cab76547179f018b987b210b09 P bb810c0459974dc6a133b30479c09c91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:12.292034 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:12.286673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:41717 (local address 127.20.92.1:46399)
0929 10:28:12.287118 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:12.287137 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:12.287267 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:12.287887 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:12.287890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:12.287911 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:12.288049 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:12.288056 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:12.289317 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:12.289320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:12.290639 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:12.290649 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:12.291575 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:12.291579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:12.291696 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:12.291735 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 10:28:12.291792 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 10:28:12.291878 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":3,"thread_start_us":130,"threads_started":1}
I20260929 10:28:12.292454 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:12.287094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:46399 (local address 127.20.92.2:41717)
0929 10:28:12.287631 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:12.287640 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:12.287658 (+ 18us) server_negotiation.cc:407] Connection header received
0929 10:28:12.287707 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:12.287710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:12.287759 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:12.287857 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:12.288211 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:12.289130 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:12.290784 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:12.291439 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:12.292108 (+ 669us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:12.292170 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:12.292206 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 10:28:12.292323 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":106,"threads_started":1}
I20260929 10:28:12.292506 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:12.286926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:40001 (local address 127.20.92.1:57783)
0929 10:28:12.287476 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:12.287491 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:12.287589 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:12.288173 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:12.288203 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:12.288219 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:12.288334 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:12.288340 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:12.289554 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:12.289561 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:12.290993 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:12.291006 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:12.292068 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:12.292074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:12.292179 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:12.292205 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 10:28:12.292278 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 10:28:12.292347 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":33,"thread_start_us":102,"threads_started":1}
I20260929 10:28:12.292996 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:12.287269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:57783 (local address 127.20.92.3:40001)
0929 10:28:12.287620 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:12.287631 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:12.287661 (+ 30us) server_negotiation.cc:407] Connection header received
0929 10:28:12.287707 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:12.287710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:12.287759 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:12.287857 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:12.288443 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:12.289411 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:12.291148 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:12.291841 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:12.291945 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:12.292002 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:12.292748 (+ 746us) server_negotiation.cc:299] Negotiation successful
0929 10:28:12.292833 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":148,"threads_started":1}
I20260929 10:28:12.292955 22875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7b4e5cab76547179f018b987b210b09" candidate_uuid: "7743196215194f4b93a5abdf914805b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb810c0459974dc6a133b30479c09c91" is_pre_election: true
I20260929 10:28:12.293443 23015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7b4e5cab76547179f018b987b210b09" candidate_uuid: "7743196215194f4b93a5abdf914805b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951cf90099f0458e9b85d9e47a6ed921" is_pre_election: true
I20260929 10:28:12.293624 23015 raft_consensus.cc:2379] T b7b4e5cab76547179f018b987b210b09 P 951cf90099f0458e9b85d9e47a6ed921 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7743196215194f4b93a5abdf914805b2 in current term 1: Already voted for candidate bb810c0459974dc6a133b30479c09c91 in this term.
I20260929 10:28:12.294129 22668 leader_election.cc:302] T b7b4e5cab76547179f018b987b210b09 P 7743196215194f4b93a5abdf914805b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7743196215194f4b93a5abdf914805b2; no voters: 951cf90099f0458e9b85d9e47a6ed921, bb810c0459974dc6a133b30479c09c91
I20260929 10:28:12.294652 23180 raft_consensus.cc:2726] T b7b4e5cab76547179f018b987b210b09 P 7743196215194f4b93a5abdf914805b2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 10:28:12.294560 22577 catalog_manager.cc:5495] T b7b4e5cab76547179f018b987b210b09 P bb810c0459974dc6a133b30479c09c91 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb810c0459974dc6a133b30479c09c91 (127.20.92.2). New cstate: current_term: 1 leader_uuid: "bb810c0459974dc6a133b30479c09c91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } health_report { overall_health: UNKNOWN } } }
I20260929 10:28:12.337069 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:12.333930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59342 (local address 127.20.92.2:41717)
0929 10:28:12.334077 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:12.334081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:12.334098 (+ 17us) server_negotiation.cc:407] Connection header received
0929 10:28:12.334187 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:12.334189 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:12.334241 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:12.334324 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:28:12.334626 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:12.335136 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:12.336060 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:12.336303 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:12.336387 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:12.336663 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:28:12.336739 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:28:12.336800 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:12.336912 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 10:28:12.336958 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 10:28:12.343609 23015 raft_consensus.cc:1261] T b7b4e5cab76547179f018b987b210b09 P 951cf90099f0458e9b85d9e47a6ed921 [term 1 FOLLOWER]: Refusing update from remote peer bb810c0459974dc6a133b30479c09c91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:12.343641 22734 raft_consensus.cc:1261] T b7b4e5cab76547179f018b987b210b09 P 7743196215194f4b93a5abdf914805b2 [term 1 FOLLOWER]: Refusing update from remote peer bb810c0459974dc6a133b30479c09c91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:12.344079 23158 consensus_queue.cc:1035] T b7b4e5cab76547179f018b987b210b09 P bb810c0459974dc6a133b30479c09c91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:28:12.344228 23156 consensus_queue.cc:1035] T b7b4e5cab76547179f018b987b210b09 P bb810c0459974dc6a133b30479c09c91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:28:12.377511 22576 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:28:12.377660 22576 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:28:12.379904 22576 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac1e7544669a4d08bb861ec2c8cfd64a: Sending DeleteTablet for 3 replicas of tablet b7b4e5cab76547179f018b987b210b09
I20260929 10:28:12.380823 22855 tablet_service.cc:1514] Processing DeleteTablet for tablet b7b4e5cab76547179f018b987b210b09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:12 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:58410
I20260929 10:28:12.380823 22715 tablet_service.cc:1514] Processing DeleteTablet for tablet b7b4e5cab76547179f018b987b210b09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:12 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:34322
I20260929 10:28:12.381052 23165 tablet_replica.cc:330] T b7b4e5cab76547179f018b987b210b09 P bb810c0459974dc6a133b30479c09c91: stopping tablet replica
I20260929 10:28:12.381054 23163 tablet_replica.cc:330] T b7b4e5cab76547179f018b987b210b09 P 7743196215194f4b93a5abdf914805b2: stopping tablet replica
I20260929 10:28:12.381069 22994 tablet_service.cc:1514] Processing DeleteTablet for tablet b7b4e5cab76547179f018b987b210b09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:12 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:39858
I20260929 10:28:12.381175 23165 raft_consensus.cc:2229] T b7b4e5cab76547179f018b987b210b09 P bb810c0459974dc6a133b30479c09c91 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:28:12.381175 23163 raft_consensus.cc:2229] T b7b4e5cab76547179f018b987b210b09 P 7743196215194f4b93a5abdf914805b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:28:12.381261 23164 tablet_replica.cc:330] T b7b4e5cab76547179f018b987b210b09 P 951cf90099f0458e9b85d9e47a6ed921: stopping tablet replica
I20260929 10:28:12.381383 23163 raft_consensus.cc:2258] T b7b4e5cab76547179f018b987b210b09 P 7743196215194f4b93a5abdf914805b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:12.381399 23164 raft_consensus.cc:2229] T b7b4e5cab76547179f018b987b210b09 P 951cf90099f0458e9b85d9e47a6ed921 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:28:12.381487 23165 raft_consensus.cc:2258] T b7b4e5cab76547179f018b987b210b09 P bb810c0459974dc6a133b30479c09c91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:12.381392 22576 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 10:28:12.381515 23164 raft_consensus.cc:2258] T b7b4e5cab76547179f018b987b210b09 P 951cf90099f0458e9b85d9e47a6ed921 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 10:28:12.381906 22576 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:28:12.382339 23165 ts_tablet_manager.cc:1905] T b7b4e5cab76547179f018b987b210b09 P bb810c0459974dc6a133b30479c09c91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:12.382349 23163 ts_tablet_manager.cc:1905] T b7b4e5cab76547179f018b987b210b09 P 7743196215194f4b93a5abdf914805b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:12.382375 23164 ts_tablet_manager.cc:1905] T b7b4e5cab76547179f018b987b210b09 P 951cf90099f0458e9b85d9e47a6ed921: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:12.385569 23165 ts_tablet_manager.cc:1918] T b7b4e5cab76547179f018b987b210b09 P bb810c0459974dc6a133b30479c09c91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:28:12.385771 23165 log.cc:1196] T b7b4e5cab76547179f018b987b210b09 P bb810c0459974dc6a133b30479c09c91: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-1/wal/wals/b7b4e5cab76547179f018b987b210b09
I20260929 10:28:12.385841 23164 ts_tablet_manager.cc:1918] T b7b4e5cab76547179f018b987b210b09 P 951cf90099f0458e9b85d9e47a6ed921: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:28:12.385952 23164 log.cc:1196] T b7b4e5cab76547179f018b987b210b09 P 951cf90099f0458e9b85d9e47a6ed921: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-2/wal/wals/b7b4e5cab76547179f018b987b210b09
I20260929 10:28:12.386126 23163 ts_tablet_manager.cc:1918] T b7b4e5cab76547179f018b987b210b09 P 7743196215194f4b93a5abdf914805b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:28:12.386266 23163 log.cc:1196] T b7b4e5cab76547179f018b987b210b09 P 7743196215194f4b93a5abdf914805b2: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-0/wal/wals/b7b4e5cab76547179f018b987b210b09
I20260929 10:28:12.386261 23164 ts_tablet_manager.cc:1939] T b7b4e5cab76547179f018b987b210b09 P 951cf90099f0458e9b85d9e47a6ed921: Deleting consensus metadata
I20260929 10:28:12.386257 23165 ts_tablet_manager.cc:1939] T b7b4e5cab76547179f018b987b210b09 P bb810c0459974dc6a133b30479c09c91: Deleting consensus metadata
I20260929 10:28:12.386618 23163 ts_tablet_manager.cc:1939] T b7b4e5cab76547179f018b987b210b09 P 7743196215194f4b93a5abdf914805b2: Deleting consensus metadata
I20260929 10:28:12.387241 22564 catalog_manager.cc:4841] TS 951cf90099f0458e9b85d9e47a6ed921 (127.20.92.3:40001): tablet b7b4e5cab76547179f018b987b210b09 (table test-table [id=de666e28996b491f877b89b05e2600ee]) successfully deleted
I20260929 10:28:12.387387 22562 catalog_manager.cc:4841] TS bb810c0459974dc6a133b30479c09c91 (127.20.92.2:41717): tablet b7b4e5cab76547179f018b987b210b09 (table test-table [id=de666e28996b491f877b89b05e2600ee]) successfully deleted
I20260929 10:28:12.388227 22563 catalog_manager.cc:4841] TS 7743196215194f4b93a5abdf914805b2 (127.20.92.1:44555): tablet b7b4e5cab76547179f018b987b210b09 (table test-table [id=de666e28996b491f877b89b05e2600ee]) successfully deleted
I20260929 10:28:12.388223 22715 tablet_service.cc:1467] Processing CreateTablet for tablet c6663b9e97de45059f6e77f115261e0a (DEFAULT_TABLE table=test-table [id=558c4c744f304c73b5233108ac9967f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:12.388307 22855 tablet_service.cc:1467] Processing CreateTablet for tablet c6663b9e97de45059f6e77f115261e0a (DEFAULT_TABLE table=test-table [id=558c4c744f304c73b5233108ac9967f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:12.388574 22855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6663b9e97de45059f6e77f115261e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:12.388579 22715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6663b9e97de45059f6e77f115261e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:12.390138 22994 tablet_service.cc:1467] Processing CreateTablet for tablet c6663b9e97de45059f6e77f115261e0a (DEFAULT_TABLE table=test-table [id=558c4c744f304c73b5233108ac9967f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:12.390463 22994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6663b9e97de45059f6e77f115261e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:12.390873 23176 tablet_bootstrap.cc:492] T c6663b9e97de45059f6e77f115261e0a P bb810c0459974dc6a133b30479c09c91: Bootstrap starting.
I20260929 10:28:12.391064 23174 tablet_bootstrap.cc:492] T c6663b9e97de45059f6e77f115261e0a P 7743196215194f4b93a5abdf914805b2: Bootstrap starting.
I20260929 10:28:12.391732 23176 tablet_bootstrap.cc:654] T c6663b9e97de45059f6e77f115261e0a P bb810c0459974dc6a133b30479c09c91: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:12.391978 23174 tablet_bootstrap.cc:654] T c6663b9e97de45059f6e77f115261e0a P 7743196215194f4b93a5abdf914805b2: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:12.392908 23176 tablet_bootstrap.cc:492] T c6663b9e97de45059f6e77f115261e0a P bb810c0459974dc6a133b30479c09c91: No bootstrap required, opened a new log
I20260929 10:28:12.393016 23176 ts_tablet_manager.cc:1397] T c6663b9e97de45059f6e77f115261e0a P bb810c0459974dc6a133b30479c09c91: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:28:12.393265 23174 tablet_bootstrap.cc:492] T c6663b9e97de45059f6e77f115261e0a P 7743196215194f4b93a5abdf914805b2: No bootstrap required, opened a new log
I20260929 10:28:12.393344 23174 ts_tablet_manager.cc:1397] T c6663b9e97de45059f6e77f115261e0a P 7743196215194f4b93a5abdf914805b2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 10:28:12.393489 23176 raft_consensus.cc:348] T c6663b9e97de45059f6e77f115261e0a P bb810c0459974dc6a133b30479c09c91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:12.393612 23176 raft_consensus.cc:374] T c6663b9e97de45059f6e77f115261e0a P bb810c0459974dc6a133b30479c09c91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:12.393648 23176 raft_consensus.cc:729] T c6663b9e97de45059f6e77f115261e0a P bb810c0459974dc6a133b30479c09c91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb810c0459974dc6a133b30479c09c91, State: Initialized, Role: FOLLOWER
I20260929 10:28:12.393817 23176 consensus_queue.cc:260] T c6663b9e97de45059f6e77f115261e0a P bb810c0459974dc6a133b30479c09c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:12.393846 23174 raft_consensus.cc:348] T c6663b9e97de45059f6e77f115261e0a P 7743196215194f4b93a5abdf914805b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:12.393962 23174 raft_consensus.cc:374] T c6663b9e97de45059f6e77f115261e0a P 7743196215194f4b93a5abdf914805b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:12.393997 23174 raft_consensus.cc:729] T c6663b9e97de45059f6e77f115261e0a P 7743196215194f4b93a5abdf914805b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7743196215194f4b93a5abdf914805b2, State: Initialized, Role: FOLLOWER
I20260929 10:28:12.394127 23176 ts_tablet_manager.cc:1428] T c6663b9e97de45059f6e77f115261e0a P bb810c0459974dc6a133b30479c09c91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:28:12.394124 23174 consensus_queue.cc:260] T c6663b9e97de45059f6e77f115261e0a P 7743196215194f4b93a5abdf914805b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:12.394428 23174 ts_tablet_manager.cc:1428] T c6663b9e97de45059f6e77f115261e0a P 7743196215194f4b93a5abdf914805b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:28:12.397681 23175 tablet_bootstrap.cc:492] T c6663b9e97de45059f6e77f115261e0a P 951cf90099f0458e9b85d9e47a6ed921: Bootstrap starting.
I20260929 10:28:12.398592 23175 tablet_bootstrap.cc:654] T c6663b9e97de45059f6e77f115261e0a P 951cf90099f0458e9b85d9e47a6ed921: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:12.399866 23175 tablet_bootstrap.cc:492] T c6663b9e97de45059f6e77f115261e0a P 951cf90099f0458e9b85d9e47a6ed921: No bootstrap required, opened a new log
I20260929 10:28:12.399976 23175 ts_tablet_manager.cc:1397] T c6663b9e97de45059f6e77f115261e0a P 951cf90099f0458e9b85d9e47a6ed921: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:28:12.400419 23175 raft_consensus.cc:348] T c6663b9e97de45059f6e77f115261e0a P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:12.400525 23175 raft_consensus.cc:374] T c6663b9e97de45059f6e77f115261e0a P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:12.400549 23175 raft_consensus.cc:729] T c6663b9e97de45059f6e77f115261e0a P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951cf90099f0458e9b85d9e47a6ed921, State: Initialized, Role: FOLLOWER
I20260929 10:28:12.400687 23175 consensus_queue.cc:260] T c6663b9e97de45059f6e77f115261e0a P 951cf90099f0458e9b85d9e47a6ed921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:12.400934 23175 ts_tablet_manager.cc:1428] T c6663b9e97de45059f6e77f115261e0a P 951cf90099f0458e9b85d9e47a6ed921: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 10:28:12.450801 23144 raft_consensus.cc:482] T c6663b9e97de45059f6e77f115261e0a P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:28:12.451025 23144 raft_consensus.cc:504] T c6663b9e97de45059f6e77f115261e0a P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:12.451548 23144 leader_election.cc:288] T c6663b9e97de45059f6e77f115261e0a P 951cf90099f0458e9b85d9e47a6ed921 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7743196215194f4b93a5abdf914805b2 (127.20.92.1:44555), bb810c0459974dc6a133b30479c09c91 (127.20.92.2:41717)
I20260929 10:28:12.452076 22735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6663b9e97de45059f6e77f115261e0a" candidate_uuid: "951cf90099f0458e9b85d9e47a6ed921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7743196215194f4b93a5abdf914805b2" is_pre_election: true
I20260929 10:28:12.452076 22875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6663b9e97de45059f6e77f115261e0a" candidate_uuid: "951cf90099f0458e9b85d9e47a6ed921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb810c0459974dc6a133b30479c09c91" is_pre_election: true
I20260929 10:28:12.452248 22875 raft_consensus.cc:2454] T c6663b9e97de45059f6e77f115261e0a P bb810c0459974dc6a133b30479c09c91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 951cf90099f0458e9b85d9e47a6ed921 in term 0.
I20260929 10:28:12.452248 22735 raft_consensus.cc:2454] T c6663b9e97de45059f6e77f115261e0a P 7743196215194f4b93a5abdf914805b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 951cf90099f0458e9b85d9e47a6ed921 in term 0.
I20260929 10:28:12.452628 22946 leader_election.cc:302] T c6663b9e97de45059f6e77f115261e0a P 951cf90099f0458e9b85d9e47a6ed921 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7743196215194f4b93a5abdf914805b2, 951cf90099f0458e9b85d9e47a6ed921, bb810c0459974dc6a133b30479c09c91; no voters:
I20260929 10:28:12.452818 23144 raft_consensus.cc:2781] T c6663b9e97de45059f6e77f115261e0a P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:28:12.452903 23144 raft_consensus.cc:482] T c6663b9e97de45059f6e77f115261e0a P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:28:12.452941 23144 raft_consensus.cc:3037] T c6663b9e97de45059f6e77f115261e0a P 951cf90099f0458e9b85d9e47a6ed921 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:12.454273 23144 raft_consensus.cc:504] T c6663b9e97de45059f6e77f115261e0a P 951cf90099f0458e9b85d9e47a6ed921 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:12.454711 23144 leader_election.cc:288] T c6663b9e97de45059f6e77f115261e0a P 951cf90099f0458e9b85d9e47a6ed921 [CANDIDATE]: Term 1 election: Requested vote from peers 7743196215194f4b93a5abdf914805b2 (127.20.92.1:44555), bb810c0459974dc6a133b30479c09c91 (127.20.92.2:41717)
I20260929 10:28:12.455008 22735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6663b9e97de45059f6e77f115261e0a" candidate_uuid: "951cf90099f0458e9b85d9e47a6ed921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7743196215194f4b93a5abdf914805b2"
I20260929 10:28:12.455178 22735 raft_consensus.cc:3037] T c6663b9e97de45059f6e77f115261e0a P 7743196215194f4b93a5abdf914805b2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:12.455176 22875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6663b9e97de45059f6e77f115261e0a" candidate_uuid: "951cf90099f0458e9b85d9e47a6ed921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb810c0459974dc6a133b30479c09c91"
I20260929 10:28:12.455323 22875 raft_consensus.cc:3037] T c6663b9e97de45059f6e77f115261e0a P bb810c0459974dc6a133b30479c09c91 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:12.456411 22875 raft_consensus.cc:2454] T c6663b9e97de45059f6e77f115261e0a P bb810c0459974dc6a133b30479c09c91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 951cf90099f0458e9b85d9e47a6ed921 in term 1.
I20260929 10:28:12.456423 22735 raft_consensus.cc:2454] T c6663b9e97de45059f6e77f115261e0a P 7743196215194f4b93a5abdf914805b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 951cf90099f0458e9b85d9e47a6ed921 in term 1.
I20260929 10:28:12.456715 22947 leader_election.cc:302] T c6663b9e97de45059f6e77f115261e0a P 951cf90099f0458e9b85d9e47a6ed921 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7743196215194f4b93a5abdf914805b2, 951cf90099f0458e9b85d9e47a6ed921; no voters:
I20260929 10:28:12.456983 23144 raft_consensus.cc:2781] T c6663b9e97de45059f6e77f115261e0a P 951cf90099f0458e9b85d9e47a6ed921 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:12.457070 23144 raft_consensus.cc:686] T c6663b9e97de45059f6e77f115261e0a P 951cf90099f0458e9b85d9e47a6ed921 [term 1 LEADER]: Becoming Leader. State: Replica: 951cf90099f0458e9b85d9e47a6ed921, State: Running, Role: LEADER
I20260929 10:28:12.457209 23144 consensus_queue.cc:237] T c6663b9e97de45059f6e77f115261e0a P 951cf90099f0458e9b85d9e47a6ed921 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } }
I20260929 10:28:12.459002 22576 catalog_manager.cc:5495] T c6663b9e97de45059f6e77f115261e0a P 951cf90099f0458e9b85d9e47a6ed921 reported cstate change: term changed from 0 to 1, leader changed from <none> to 951cf90099f0458e9b85d9e47a6ed921 (127.20.92.3). New cstate: current_term: 1 leader_uuid: "951cf90099f0458e9b85d9e47a6ed921" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "951cf90099f0458e9b85d9e47a6ed921" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40001 } health_report { overall_health: HEALTHY } } }
I20260929 10:28:12.484371 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:12.480897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49728 (local address 127.20.92.3:40001)
0929 10:28:12.481004 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:12.481009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:12.481025 (+ 16us) server_negotiation.cc:407] Connection header received
0929 10:28:12.481072 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:12.481075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:12.481125 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:12.481210 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:28:12.481447 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:12.482402 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:12.483237 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:12.483615 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:12.483719 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:12.483929 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:28:12.484012 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:28:12.484086 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:12.484161 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 10:28:12.484223 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 10:28:12.491642 22875 raft_consensus.cc:1261] T c6663b9e97de45059f6e77f115261e0a P bb810c0459974dc6a133b30479c09c91 [term 1 FOLLOWER]: Refusing update from remote peer 951cf90099f0458e9b85d9e47a6ed921: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:12.491866 22735 raft_consensus.cc:1261] T c6663b9e97de45059f6e77f115261e0a P 7743196215194f4b93a5abdf914805b2 [term 1 FOLLOWER]: Refusing update from remote peer 951cf90099f0458e9b85d9e47a6ed921: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:12.492053 23195 consensus_queue.cc:1035] T c6663b9e97de45059f6e77f115261e0a P 951cf90099f0458e9b85d9e47a6ed921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb810c0459974dc6a133b30479c09c91" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:28:12.492319 23144 consensus_queue.cc:1035] T c6663b9e97de45059f6e77f115261e0a P 951cf90099f0458e9b85d9e47a6ed921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7743196215194f4b93a5abdf914805b2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 10:28:12.506152 23142 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 10:28:14.061542 22577 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:28:14.061707 22577 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:28:14.064141 22577 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac1e7544669a4d08bb861ec2c8cfd64a: Sending DeleteTablet for 3 replicas of tablet c6663b9e97de45059f6e77f115261e0a
I20260929 10:28:14.064749 22994 tablet_service.cc:1514] Processing DeleteTablet for tablet c6663b9e97de45059f6e77f115261e0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:14 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:39858
I20260929 10:28:14.064740 22715 tablet_service.cc:1514] Processing DeleteTablet for tablet c6663b9e97de45059f6e77f115261e0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:14 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:34322
I20260929 10:28:14.064814 22855 tablet_service.cc:1514] Processing DeleteTablet for tablet c6663b9e97de45059f6e77f115261e0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:14 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:58410
I20260929 10:28:14.065143 23219 tablet_replica.cc:330] T c6663b9e97de45059f6e77f115261e0a P bb810c0459974dc6a133b30479c09c91: stopping tablet replica
I20260929 10:28:14.065109 23217 tablet_replica.cc:330] T c6663b9e97de45059f6e77f115261e0a P 951cf90099f0458e9b85d9e47a6ed921: stopping tablet replica
I20260929 10:28:14.065200 23218 tablet_replica.cc:330] T c6663b9e97de45059f6e77f115261e0a P 7743196215194f4b93a5abdf914805b2: stopping tablet replica
I20260929 10:28:14.065295 23217 raft_consensus.cc:2229] T c6663b9e97de45059f6e77f115261e0a P 951cf90099f0458e9b85d9e47a6ed921 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:28:14.065366 23218 raft_consensus.cc:2229] T c6663b9e97de45059f6e77f115261e0a P 7743196215194f4b93a5abdf914805b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:28:14.065377 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 22650
I20260929 10:28:14.065403 23219 raft_consensus.cc:2229] T c6663b9e97de45059f6e77f115261e0a P bb810c0459974dc6a133b30479c09c91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:28:14.066262 23219 raft_consensus.cc:2258] T c6663b9e97de45059f6e77f115261e0a P bb810c0459974dc6a133b30479c09c91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:14.066385 23217 raft_consensus.cc:2258] T c6663b9e97de45059f6e77f115261e0a P 951cf90099f0458e9b85d9e47a6ed921 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:14.067071 23217 ts_tablet_manager.cc:1905] T c6663b9e97de45059f6e77f115261e0a P 951cf90099f0458e9b85d9e47a6ed921: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:14.067071 23219 ts_tablet_manager.cc:1905] T c6663b9e97de45059f6e77f115261e0a P bb810c0459974dc6a133b30479c09c91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:14.069995 23219 ts_tablet_manager.cc:1918] T c6663b9e97de45059f6e77f115261e0a P bb810c0459974dc6a133b30479c09c91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 10:28:14.069995 23217 ts_tablet_manager.cc:1918] T c6663b9e97de45059f6e77f115261e0a P 951cf90099f0458e9b85d9e47a6ed921: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 10:28:14.070083 23217 log.cc:1196] T c6663b9e97de45059f6e77f115261e0a P 951cf90099f0458e9b85d9e47a6ed921: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-2/wal/wals/c6663b9e97de45059f6e77f115261e0a
I20260929 10:28:14.070235 23219 log.cc:1196] T c6663b9e97de45059f6e77f115261e0a P bb810c0459974dc6a133b30479c09c91: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677679287822-20848-0/minicluster-data/ts-1/wal/wals/c6663b9e97de45059f6e77f115261e0a
I20260929 10:28:14.070436 23217 ts_tablet_manager.cc:1939] T c6663b9e97de45059f6e77f115261e0a P 951cf90099f0458e9b85d9e47a6ed921: Deleting consensus metadata
I20260929 10:28:14.070593 23219 ts_tablet_manager.cc:1939] T c6663b9e97de45059f6e77f115261e0a P bb810c0459974dc6a133b30479c09c91: Deleting consensus metadata
I20260929 10:28:14.071662 22562 catalog_manager.cc:4841] TS bb810c0459974dc6a133b30479c09c91 (127.20.92.2:41717): tablet c6663b9e97de45059f6e77f115261e0a (table test-table [id=558c4c744f304c73b5233108ac9967f1]) successfully deleted
W20260929 10:28:14.073251 22563 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.92.1:44555 (error 108)
I20260929 10:28:14.073316 22564 catalog_manager.cc:4841] TS 951cf90099f0458e9b85d9e47a6ed921 (127.20.92.3:40001): tablet c6663b9e97de45059f6e77f115261e0a (table test-table [id=558c4c744f304c73b5233108ac9967f1]) successfully deleted
I20260929 10:28:14.073640 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 22790
I20260929 10:28:14.076082 23221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:14.075633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:44555 (local address 127.0.0.1:59714)
0929 10:28:14.075916 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:14.075993 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.1:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":108,"threads_started":1}
W20260929 10:28:14.076285 22563 catalog_manager.cc:4584] TS 7743196215194f4b93a5abdf914805b2 (127.20.92.1:44555): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c6663b9e97de45059f6e77f115261e0a: Network error: Client connection negotiation failed: client connection to 127.20.92.1:44555: connect: Connection refused (error 111)
I20260929 10:28:14.081382 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 22930
I20260929 10:28:14.088326 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 22546
2026-09-29T10:28:14Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6600 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:28:14 dist-test-slave-9cj5 krb5kdc[23224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:28:14 dist-test-slave-9cj5 krb5kdc[23224](info): set up 2 sockets
Sep 29 10:28:14 dist-test-slave-9cj5 krb5kdc[23224](info): commencing operation
krb5kdc: starting...
W20260929 10:28:15.166698 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:28:15 dist-test-slave-9cj5 krb5kdc[23224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:28:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:28:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
I20260929 10:28:15.300938 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:42217
--webserver_interface=127.20.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:46269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:42217
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:15.413362 23237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:15.413698 23237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:15.413825 23237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:15.417920 23237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:28:15.418018 23237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:15.418057 23237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:15.418118 23237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:28:15.418172 23237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:28:15.423626 23237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:46269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:42217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:42217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:15.424772 23237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:15.425915 23237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:15.434828 23245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:15.434828 23243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:15.434991 23237 server_base.cc:1034] running on GCE node
W20260929 10:28:15.434857 23242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:15.435698 23237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:15.436858 23237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:15.438030 23237 hybrid_clock.cc:648] HybridClock initialized: now 1790677695438000 us; error 43 us; skew 500 ppm
Sep 29 10:28:15 dist-test-slave-9cj5 krb5kdc[23224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677695, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:15.441071 23237 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:28:15.442153 23237 webserver.cc:460] Webserver started at http://127.20.92.62:42917/ using document root <none> and password file <none>
I20260929 10:28:15.442694 23237 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:15.442755 23237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:15.442929 23237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:15.445053 23237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "84910c53f9a745e2bbdf419bedfbb6b3"
format_stamp: "Formatted at 2026-09-29 10:28:15 on dist-test-slave-9cj5"
server_key: "5c1eb769eec480cf65eb839b95ac7c6c"
server_key_iv: "7f139914b8da68afba2643ed62b20f32"
server_key_version: "encryptionkey@0"
I20260929 10:28:15.445600 23237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "84910c53f9a745e2bbdf419bedfbb6b3"
format_stamp: "Formatted at 2026-09-29 10:28:15 on dist-test-slave-9cj5"
server_key: "5c1eb769eec480cf65eb839b95ac7c6c"
server_key_iv: "7f139914b8da68afba2643ed62b20f32"
server_key_version: "encryptionkey@0"
I20260929 10:28:15.449571 23237 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260929 10:28:15.452337 23252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:15.453718 23237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:28:15.453969 23237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "84910c53f9a745e2bbdf419bedfbb6b3"
format_stamp: "Formatted at 2026-09-29 10:28:15 on dist-test-slave-9cj5"
server_key: "5c1eb769eec480cf65eb839b95ac7c6c"
server_key_iv: "7f139914b8da68afba2643ed62b20f32"
server_key_version: "encryptionkey@0"
I20260929 10:28:15.454090 23237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:15.467563 23237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:15.470816 23237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:15.471045 23237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:15.478618 23237 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:42217
I20260929 10:28:15.478643 23304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:42217 every 8 connection(s)
I20260929 10:28:15.479594 23237 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:28:15.482470 23305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:15.487779 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 23237
I20260929 10:28:15.487902 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:28:15.488164 20848 external_mini_cluster.cc:1428] Setting key 76349d43c4eeaae54fc1a9b1bf865646
I20260929 10:28:15.488262 23305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84910c53f9a745e2bbdf419bedfbb6b3: Bootstrap starting.
I20260929 10:28:15.490854 23305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84910c53f9a745e2bbdf419bedfbb6b3: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:15.491637 23305 log.cc:824] T 00000000000000000000000000000000 P 84910c53f9a745e2bbdf419bedfbb6b3: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:15.493908 23305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84910c53f9a745e2bbdf419bedfbb6b3: No bootstrap required, opened a new log
Sep 29 10:28:15 dist-test-slave-9cj5 krb5kdc[23224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:15.498572 23305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84910c53f9a745e2bbdf419bedfbb6b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84910c53f9a745e2bbdf419bedfbb6b3" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 42217 } }
I20260929 10:28:15.498852 23305 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84910c53f9a745e2bbdf419bedfbb6b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:15.498945 23305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84910c53f9a745e2bbdf419bedfbb6b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84910c53f9a745e2bbdf419bedfbb6b3, State: Initialized, Role: FOLLOWER
I20260929 10:28:15.499526 23305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84910c53f9a745e2bbdf419bedfbb6b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84910c53f9a745e2bbdf419bedfbb6b3" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 42217 } }
I20260929 10:28:15.499732 23305 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84910c53f9a745e2bbdf419bedfbb6b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:28:15.499799 23305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84910c53f9a745e2bbdf419bedfbb6b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:28:15.499907 23305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84910c53f9a745e2bbdf419bedfbb6b3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:15.501123 23305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84910c53f9a745e2bbdf419bedfbb6b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84910c53f9a745e2bbdf419bedfbb6b3" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 42217 } }
I20260929 10:28:15.501554 23305 leader_election.cc:302] T 00000000000000000000000000000000 P 84910c53f9a745e2bbdf419bedfbb6b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84910c53f9a745e2bbdf419bedfbb6b3; no voters:
I20260929 10:28:15.501938 23305 leader_election.cc:288] T 00000000000000000000000000000000 P 84910c53f9a745e2bbdf419bedfbb6b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:28:15.502064 23310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84910c53f9a745e2bbdf419bedfbb6b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:15.502319 23310 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84910c53f9a745e2bbdf419bedfbb6b3 [term 1 LEADER]: Becoming Leader. State: Replica: 84910c53f9a745e2bbdf419bedfbb6b3, State: Running, Role: LEADER
I20260929 10:28:15.502702 23310 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84910c53f9a745e2bbdf419bedfbb6b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84910c53f9a745e2bbdf419bedfbb6b3" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 42217 } }
I20260929 10:28:15.503329 23305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84910c53f9a745e2bbdf419bedfbb6b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:28:15.504405 23311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84910c53f9a745e2bbdf419bedfbb6b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84910c53f9a745e2bbdf419bedfbb6b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84910c53f9a745e2bbdf419bedfbb6b3" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 42217 } } }
I20260929 10:28:15.504465 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:15.489574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55606 (local address 127.20.92.62:42217)
0929 10:28:15.490023 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:15.490039 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:15.490073 (+ 34us) server_negotiation.cc:407] Connection header received
0929 10:28:15.490689 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:15.490705 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:15.491104 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:15.491447 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:15.492332 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:15.493738 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:15.494674 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:15.495368 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:15.495540 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:15.498363 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:15.498370 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:15.498387 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:15.498419 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:15.500598 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:15.501089 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:15.501098 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:15.501107 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:15.501288 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:15.501555 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:15.501558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:15.501561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:15.502156 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:15.502949 (+ 793us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:15.503103 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 10:28:15.503409 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":159,"threads_started":1}
I20260929 10:28:15.504525 23311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84910c53f9a745e2bbdf419bedfbb6b3 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:15.504865 23312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84910c53f9a745e2bbdf419bedfbb6b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84910c53f9a745e2bbdf419bedfbb6b3. Latest consensus state: current_term: 1 leader_uuid: "84910c53f9a745e2bbdf419bedfbb6b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84910c53f9a745e2bbdf419bedfbb6b3" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 42217 } } }
I20260929 10:28:15.504949 23312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84910c53f9a745e2bbdf419bedfbb6b3 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:15.505589 23319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:28:15.509111 23319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:28:15.515012 23319 catalog_manager.cc:1347] Generated new cluster ID: bdca392e702e4ebd97d91566eac09952
I20260929 10:28:15.515100 23319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:28:15.527128 23319 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:28:15.528119 23319 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:28:15.535080 23319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84910c53f9a745e2bbdf419bedfbb6b3: Generated new TSK 0
I20260929 10:28:15.535840 23319 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
I20260929 10:28:15.573096 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:0
--local_ip_for_outbound_sockets=127.20.92.1
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:42217
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:46269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:15.702699 23331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:15.702975 23331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:15.703073 23331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:15.707432 23331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:15.707501 23331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:15.707563 23331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:28:15.712975 23331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:46269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:42217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:15.714051 23331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:15.715121 23331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:15.724749 23339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:15.724758 23336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:15.724784 23337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:15.724922 23331 server_base.cc:1034] running on GCE node
I20260929 10:28:15.725425 23331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:15.726147 23331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:15.727345 23331 hybrid_clock.cc:648] HybridClock initialized: now 1790677695727324 us; error 26 us; skew 500 ppm
Sep 29 10:28:15 dist-test-slave-9cj5 krb5kdc[23224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677695, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:15.730484 23331 init.cc:377] Logged in from keytab as kudu/127.20.92.1@KRBTEST.COM (short username kudu)
I20260929 10:28:15.731484 23331 webserver.cc:460] Webserver started at http://127.20.92.1:36147/ using document root <none> and password file <none>
I20260929 10:28:15.732059 23331 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:15.732120 23331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:15.732342 23331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:15.734611 23331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/data/instance:
uuid: "6b30f7b39fb6494bb6b2cf62d349d5ca"
format_stamp: "Formatted at 2026-09-29 10:28:15 on dist-test-slave-9cj5"
server_key: "41970788d79c2fa0e223c491af08ed81"
server_key_iv: "6ae5a7d20522b7fc68009424ef1aa3a6"
server_key_version: "encryptionkey@0"
I20260929 10:28:15.735159 23331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance:
uuid: "6b30f7b39fb6494bb6b2cf62d349d5ca"
format_stamp: "Formatted at 2026-09-29 10:28:15 on dist-test-slave-9cj5"
server_key: "41970788d79c2fa0e223c491af08ed81"
server_key_iv: "6ae5a7d20522b7fc68009424ef1aa3a6"
server_key_version: "encryptionkey@0"
I20260929 10:28:15.739289 23331 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 10:28:15.741959 23346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:15.743335 23331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:28:15.743445 23331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "6b30f7b39fb6494bb6b2cf62d349d5ca"
format_stamp: "Formatted at 2026-09-29 10:28:15 on dist-test-slave-9cj5"
server_key: "41970788d79c2fa0e223c491af08ed81"
server_key_iv: "6ae5a7d20522b7fc68009424ef1aa3a6"
server_key_version: "encryptionkey@0"
I20260929 10:28:15.743546 23331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:15.775251 23331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:15.778580 23331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:15.778741 23331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:15.779294 23331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:15.780148 23331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:15.780193 23331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:15.780262 23331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:15.780295 23331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:15.789770 23331 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:38189
I20260929 10:28:15.789798 23459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:38189 every 8 connection(s)
I20260929 10:28:15.790655 23331 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
Sep 29 10:28:15 dist-test-slave-9cj5 krb5kdc[23224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677695, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:15.800954 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 23331
I20260929 10:28:15.801071 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance
I20260929 10:28:15.801381 20848 external_mini_cluster.cc:1428] Setting key 6bbd2da2fdb6058ac809eebb8522c7ab
I20260929 10:28:15.806478 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:15.792560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:45649 (local address 127.20.92.62:42217)
0929 10:28:15.792740 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:15.792746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:15.793593 (+ 847us) server_negotiation.cc:407] Connection header received
0929 10:28:15.794965 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:15.794967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:15.795013 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:15.795087 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:15.796418 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:15.796984 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:15.798338 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:15.798567 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:15.798651 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:15.801540 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:15.801543 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:15.801546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:15.801575 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:15.803696 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:15.804410 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:15.804417 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:15.804422 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:15.804564 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:15.805061 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:15.805068 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:15.805072 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:15.805305 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:15.805391 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:15.806179 (+ 788us) server_negotiation.cc:299] Negotiation successful
0929 10:28:15.806309 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 10:28:15.807559 23462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:15.792956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:42217 (local address 127.20.92.1:45649)
0929 10:28:15.793439 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:15.793480 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:15.794761 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:15.795317 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:15.795324 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:15.795756 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:15.796278 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:15.796292 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:15.797122 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:15.797125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:15.798218 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:15.798227 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:15.798739 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:15.798742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:15.798880 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:15.799526 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:15.799542 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:15.801333 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:15.803839 (+ 2506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:15.803845 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:15.803870 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:15.804250 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:15.804692 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:15.804698 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:15.804703 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:15.804921 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:15.805477 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:15.805484 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:15.805862 (+ 378us) client_negotiation.cc:769] Sending connection context
0929 10:28:15.806120 (+ 258us) client_negotiation.cc:240] Negotiation successful
0929 10:28:15.806467 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":106,"threads_started":1}
I20260929 10:28:15.809864 23460 heartbeater.cc:347] Connected to a master server at 127.20.92.62:42217
I20260929 10:28:15.810222 23460 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:15.810834 23460 heartbeater.cc:511] Master 127.20.92.62:42217 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.92.2@KRBTEST.COM; defaulting to no policy
I20260929 10:28:15.812567 23269 ts_manager.cc:194] Registered new tserver with Master: 6b30f7b39fb6494bb6b2cf62d349d5ca (127.20.92.1:38189)
I20260929 10:28:15.813968 23269 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.1@KRBTEST.COM'} at 127.20.92.1:45649
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
I20260929 10:28:15.834079 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:0
--local_ip_for_outbound_sockets=127.20.92.2
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:42217
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:46269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:15.943819 23465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:15.944085 23465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:15.944128 23465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:15.948360 23465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:15.948421 23465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:15.948520 23465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:28:15.953795 23465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:46269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:42217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:15.954854 23465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:15.955840 23465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:15.965726 23473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:15.965931 23470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:15.966002 23465 server_base.cc:1034] running on GCE node
W20260929 10:28:15.965909 23471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:15.966475 23465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:15.967046 23465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:15.968209 23465 hybrid_clock.cc:648] HybridClock initialized: now 1790677695968168 us; error 50 us; skew 500 ppm
Sep 29 10:28:15 dist-test-slave-9cj5 krb5kdc[23224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677695, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:15.971019 23465 init.cc:377] Logged in from keytab as kudu/127.20.92.2@KRBTEST.COM (short username kudu)
I20260929 10:28:15.971935 23465 webserver.cc:460] Webserver started at http://127.20.92.2:35489/ using document root <none> and password file <none>
I20260929 10:28:15.972498 23465 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:15.972564 23465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:15.972739 23465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:15.974992 23465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/data/instance:
uuid: "a177263c847045f0b5b2c9c7ccf8e6bd"
format_stamp: "Formatted at 2026-09-29 10:28:15 on dist-test-slave-9cj5"
server_key: "fd24d52a17d2dae6ececb499dabc19de"
server_key_iv: "7fcd41dfb4cb2bf1149a9d9102bb84ef"
server_key_version: "encryptionkey@0"
I20260929 10:28:15.975543 23465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance:
uuid: "a177263c847045f0b5b2c9c7ccf8e6bd"
format_stamp: "Formatted at 2026-09-29 10:28:15 on dist-test-slave-9cj5"
server_key: "fd24d52a17d2dae6ececb499dabc19de"
server_key_iv: "7fcd41dfb4cb2bf1149a9d9102bb84ef"
server_key_version: "encryptionkey@0"
I20260929 10:28:15.979523 23465 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:28:15.982129 23480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:15.983398 23465 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 10:28:15.983496 23465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "a177263c847045f0b5b2c9c7ccf8e6bd"
format_stamp: "Formatted at 2026-09-29 10:28:15 on dist-test-slave-9cj5"
server_key: "fd24d52a17d2dae6ececb499dabc19de"
server_key_iv: "7fcd41dfb4cb2bf1149a9d9102bb84ef"
server_key_version: "encryptionkey@0"
I20260929 10:28:15.983582 23465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:16.001359 23465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:16.004801 23465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:16.005056 23465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:16.005674 23465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:16.006582 23465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:16.006664 23465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:16.006731 23465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:16.006783 23465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:16.015877 23465 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:37945
I20260929 10:28:16.015933 23593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:37945 every 8 connection(s)
I20260929 10:28:16.016853 23465 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
I20260929 10:28:16.020583 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 23465
I20260929 10:28:16.020699 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance
I20260929 10:28:16.020975 20848 external_mini_cluster.cc:1428] Setting key d70eff003df8f0ccc6c69eb3f09633f4
Sep 29 10:28:16 dist-test-slave-9cj5 krb5kdc[23224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677695, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.92.3@KRBTEST.COM; defaulting to no policy
I20260929 10:28:16.032871 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:16.018872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:58975 (local address 127.20.92.62:42217)
0929 10:28:16.019025 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:16.019029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:16.020193 (+ 1164us) server_negotiation.cc:407] Connection header received
0929 10:28:16.021381 (+ 1188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:16.021385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:16.021451 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:16.021561 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:16.023364 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:16.023877 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:16.025147 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:16.025549 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:16.025677 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:16.028715 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:16.028721 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:16.028726 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:16.028758 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:16.030636 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:16.031368 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:16.031371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:16.031373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:16.031445 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:16.031750 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:16.031752 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:16.031753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:16.031971 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:16.032090 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:16.032601 (+ 511us) server_negotiation.cc:299] Negotiation successful
0929 10:28:16.032709 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 10:28:16.033842 23596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:16.019324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:42217 (local address 127.20.92.2:58975)
0929 10:28:16.020017 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:16.020060 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:16.021117 (+ 1057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:16.021966 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:16.021973 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:16.022453 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:16.023164 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:16.023189 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:16.024018 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:16.024024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:16.025011 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:16.025021 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:16.025700 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:16.025702 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:16.025940 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:16.026631 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:16.026651 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:16.028514 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:16.030815 (+ 2301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:16.030820 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:16.030836 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:16.031257 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:16.031531 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:16.031534 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:16.031535 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:16.031667 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:16.032117 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:16.032125 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:16.032390 (+ 265us) client_negotiation.cc:769] Sending connection context
0929 10:28:16.032556 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 10:28:16.032816 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":130,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.3@KRBTEST.COM" created.
I20260929 10:28:16.035135 23594 heartbeater.cc:347] Connected to a master server at 127.20.92.62:42217
I20260929 10:28:16.035408 23594 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:16.036092 23594 heartbeater.cc:511] Master 127.20.92.62:42217 requested a full tablet report, sending...
I20260929 10:28:16.037313 23269 ts_manager.cc:194] Registered new tserver with Master: a177263c847045f0b5b2c9c7ccf8e6bd (127.20.92.2:37945)
I20260929 10:28:16.038038 23269 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.2@KRBTEST.COM'} at 127.20.92.2:58975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
I20260929 10:28:16.051352 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:0
--local_ip_for_outbound_sockets=127.20.92.3
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:42217
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:46269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:16.161732 23599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:16.162115 23599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:16.162189 23599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:16.166338 23599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:16.166417 23599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:16.166504 23599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:28:16.171835 23599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:46269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:42217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:16.172909 23599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:16.174016 23599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:16.183790 23604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:16.183916 23599 server_base.cc:1034] running on GCE node
W20260929 10:28:16.183833 23607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:16.183809 23605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:16.184477 23599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:16.185196 23599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:16.186417 23599 hybrid_clock.cc:648] HybridClock initialized: now 1790677696186395 us; error 30 us; skew 500 ppm
Sep 29 10:28:16 dist-test-slave-9cj5 krb5kdc[23224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677696, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:16.189976 23599 init.cc:377] Logged in from keytab as kudu/127.20.92.3@KRBTEST.COM (short username kudu)
I20260929 10:28:16.190860 23599 webserver.cc:460] Webserver started at http://127.20.92.3:42757/ using document root <none> and password file <none>
I20260929 10:28:16.191427 23599 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:16.191516 23599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:16.191740 23599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:16.193825 23599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/data/instance:
uuid: "ce1a77cc9a894a018bb01211107ff8d3"
format_stamp: "Formatted at 2026-09-29 10:28:16 on dist-test-slave-9cj5"
server_key: "bd1714b21e96ed5b91c72a0c50594a01"
server_key_iv: "4f7e3b1a3f3d10f4668cbb7b999b14c4"
server_key_version: "encryptionkey@0"
I20260929 10:28:16.194424 23599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance:
uuid: "ce1a77cc9a894a018bb01211107ff8d3"
format_stamp: "Formatted at 2026-09-29 10:28:16 on dist-test-slave-9cj5"
server_key: "bd1714b21e96ed5b91c72a0c50594a01"
server_key_iv: "4f7e3b1a3f3d10f4668cbb7b999b14c4"
server_key_version: "encryptionkey@0"
I20260929 10:28:16.198319 23599 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 10:28:16.200847 23614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:16.202205 23599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:28:16.202345 23599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "ce1a77cc9a894a018bb01211107ff8d3"
format_stamp: "Formatted at 2026-09-29 10:28:16 on dist-test-slave-9cj5"
server_key: "bd1714b21e96ed5b91c72a0c50594a01"
server_key_iv: "4f7e3b1a3f3d10f4668cbb7b999b14c4"
server_key_version: "encryptionkey@0"
I20260929 10:28:16.202472 23599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:16.223359 23599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:16.227021 23599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:16.227240 23599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:16.227875 23599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:16.228745 23599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:16.228819 23599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:16.228892 23599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:16.228926 23599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:16.238783 23599 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:33513
I20260929 10:28:16.238857 23727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:33513 every 8 connection(s)
I20260929 10:28:16.239749 23599 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:28:16.248132 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 23599
I20260929 10:28:16.248265 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance
I20260929 10:28:16.248590 20848 external_mini_cluster.cc:1428] Setting key 973d3e9834bcc771bbed00267a73602b
Sep 29 10:28:16 dist-test-slave-9cj5 krb5kdc[23224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677696, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:16.254913 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:16.241692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:40183 (local address 127.20.92.62:42217)
0929 10:28:16.241896 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:16.241900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:16.242671 (+ 771us) server_negotiation.cc:407] Connection header received
0929 10:28:16.244023 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:16.244026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:16.244089 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:16.244171 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:16.245359 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:16.245983 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:16.247136 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:16.247371 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:16.247497 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:16.250641 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:16.250645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:16.250648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:16.250678 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:16.252479 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:16.253087 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:16.253091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:16.253093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:16.253193 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:16.253579 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:16.253584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:16.253586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:16.253896 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:16.254020 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:16.254579 (+ 559us) server_negotiation.cc:299] Negotiation successful
0929 10:28:16.254714 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 10:28:16.255775 23730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:16.242097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:42217 (local address 127.20.92.3:40183)
0929 10:28:16.242522 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:16.242564 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:16.243794 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:16.244345 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:16.244352 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:16.244713 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:16.245195 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:16.245209 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:16.246114 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:16.246122 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:16.247006 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:16.247016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:16.247551 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:16.247555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:16.247758 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:16.248473 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:16.248498 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:16.250465 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:16.252629 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:16.252637 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:16.252655 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:16.252968 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:16.253308 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:16.253311 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:16.253313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:16.253472 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:16.254016 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:16.254024 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:16.254315 (+ 291us) client_negotiation.cc:769] Sending connection context
0929 10:28:16.254523 (+ 208us) client_negotiation.cc:240] Negotiation successful
0929 10:28:16.254884 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":102,"threads_started":1}
I20260929 10:28:16.257026 23728 heartbeater.cc:347] Connected to a master server at 127.20.92.62:42217
I20260929 10:28:16.257299 23728 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:16.257967 23728 heartbeater.cc:511] Master 127.20.92.62:42217 requested a full tablet report, sending...
I20260929 10:28:16.259097 23269 ts_manager.cc:194] Registered new tserver with Master: ce1a77cc9a894a018bb01211107ff8d3 (127.20.92.3:33513)
I20260929 10:28:16.259634 23269 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.3@KRBTEST.COM'} at 127.20.92.3:40183
I20260929 10:28:16.263309 20848 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 10:28:16.276721 23308 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:16.275084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55618 (local address 127.20.92.62:42217)
0929 10:28:16.275255 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:16.275259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:16.275301 (+ 42us) server_negotiation.cc:407] Connection header received
0929 10:28:16.275502 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:16.275506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:16.275555 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:16.275635 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:16.276526 (+ 891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55618: BlockingRecv error: recv got EOF from 127.0.0.1:55618 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 10:28:16.277925 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 23331
I20260929 10:28:16.283558 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 23465
I20260929 10:28:16.289026 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 23599
I20260929 10:28:16.294497 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 23237
2026-09-29T10:28:16Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2204 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:28:16 dist-test-slave-9cj5 krb5kdc[23740](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:28:16 dist-test-slave-9cj5 krb5kdc[23740](info): set up 2 sockets
Sep 29 10:28:16 dist-test-slave-9cj5 krb5kdc[23740](info): commencing operation
krb5kdc: starting...
W20260929 10:28:17.342648 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:28:17 dist-test-slave-9cj5 krb5kdc[23740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:28:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:28:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
I20260929 10:28:17.470525 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:42015
--webserver_interface=127.20.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:35063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:42015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:17.582497 23753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:17.582809 23753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:17.582870 23753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:17.586918 23753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:28:17.586997 23753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:17.587033 23753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:17.587090 23753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:28:17.587127 23753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:28:17.593016 23753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:35063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:42015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:42015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:17.594249 23753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:17.595352 23753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:17.604457 23758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:17.604453 23761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:17.604449 23759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:17.604712 23753 server_base.cc:1034] running on GCE node
I20260929 10:28:17.605363 23753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:17.606597 23753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:17.607802 23753 hybrid_clock.cc:648] HybridClock initialized: now 1790677697607771 us; error 46 us; skew 500 ppm
Sep 29 10:28:17 dist-test-slave-9cj5 krb5kdc[23740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677697, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:17.610671 23753 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:28:17.611652 23753 webserver.cc:460] Webserver started at http://127.20.92.62:46459/ using document root <none> and password file <none>
I20260929 10:28:17.612175 23753 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:17.612232 23753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:17.612397 23753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:17.614447 23753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "d2619af372df461298b116c9c114bb50"
format_stamp: "Formatted at 2026-09-29 10:28:17 on dist-test-slave-9cj5"
server_key: "a11b1cf0e71e8470971b912d2b3e52d6"
server_key_iv: "598fccbf29234dfd7f22ac0214bc84c5"
server_key_version: "encryptionkey@0"
I20260929 10:28:17.614964 23753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "d2619af372df461298b116c9c114bb50"
format_stamp: "Formatted at 2026-09-29 10:28:17 on dist-test-slave-9cj5"
server_key: "a11b1cf0e71e8470971b912d2b3e52d6"
server_key_iv: "598fccbf29234dfd7f22ac0214bc84c5"
server_key_version: "encryptionkey@0"
I20260929 10:28:17.618933 23753 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 10:28:17.621874 23768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:17.623299 23753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:28:17.623416 23753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "d2619af372df461298b116c9c114bb50"
format_stamp: "Formatted at 2026-09-29 10:28:17 on dist-test-slave-9cj5"
server_key: "a11b1cf0e71e8470971b912d2b3e52d6"
server_key_iv: "598fccbf29234dfd7f22ac0214bc84c5"
server_key_version: "encryptionkey@0"
I20260929 10:28:17.623569 23753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:17.640132 23753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:17.643493 23753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:17.643709 23753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:17.651217 23753 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:42015
I20260929 10:28:17.651533 23820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:42015 every 8 connection(s)
I20260929 10:28:17.652228 23753 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:28:17.655097 23821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:17.657203 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 23753
I20260929 10:28:17.657332 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:28:17.657595 20848 external_mini_cluster.cc:1428] Setting key 8b3136dacd34ae5abd31bb07011478fc
I20260929 10:28:17.661325 23821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2619af372df461298b116c9c114bb50: Bootstrap starting.
I20260929 10:28:17.663872 23821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2619af372df461298b116c9c114bb50: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:17.664678 23821 log.cc:824] T 00000000000000000000000000000000 P d2619af372df461298b116c9c114bb50: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:17.666814 23821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2619af372df461298b116c9c114bb50: No bootstrap required, opened a new log
Sep 29 10:28:17 dist-test-slave-9cj5 krb5kdc[23740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:17.670146 23821 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2619af372df461298b116c9c114bb50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2619af372df461298b116c9c114bb50" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 42015 } }
I20260929 10:28:17.670414 23821 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2619af372df461298b116c9c114bb50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:17.670504 23821 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2619af372df461298b116c9c114bb50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2619af372df461298b116c9c114bb50, State: Initialized, Role: FOLLOWER
I20260929 10:28:17.671018 23821 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2619af372df461298b116c9c114bb50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2619af372df461298b116c9c114bb50" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 42015 } }
I20260929 10:28:17.671175 23821 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2619af372df461298b116c9c114bb50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:28:17.671262 23821 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2619af372df461298b116c9c114bb50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:28:17.671387 23821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2619af372df461298b116c9c114bb50 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:17.672659 23821 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2619af372df461298b116c9c114bb50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2619af372df461298b116c9c114bb50" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 42015 } }
I20260929 10:28:17.673024 23821 leader_election.cc:302] T 00000000000000000000000000000000 P d2619af372df461298b116c9c114bb50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2619af372df461298b116c9c114bb50; no voters:
I20260929 10:28:17.673409 23821 leader_election.cc:288] T 00000000000000000000000000000000 P d2619af372df461298b116c9c114bb50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:28:17.673547 23826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2619af372df461298b116c9c114bb50 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:17.673839 23826 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2619af372df461298b116c9c114bb50 [term 1 LEADER]: Becoming Leader. State: Replica: d2619af372df461298b116c9c114bb50, State: Running, Role: LEADER
I20260929 10:28:17.674202 23824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:17.659161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50976 (local address 127.20.92.62:42015)
0929 10:28:17.659599 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:17.659622 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:17.659659 (+ 37us) server_negotiation.cc:407] Connection header received
0929 10:28:17.660399 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:17.660417 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:17.660761 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:17.661065 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:17.662053 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:17.663077 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:17.664297 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:17.664919 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:17.665105 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:17.667913 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:17.667922 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:17.667948 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:17.667978 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:17.670775 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:17.671346 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:17.671352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:17.671358 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:17.671518 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:17.671819 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:17.671823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:17.671824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:17.672359 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:17.672578 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:17.672831 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 10:28:17.673142 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":133,"threads_started":1}
I20260929 10:28:17.674229 23826 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2619af372df461298b116c9c114bb50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2619af372df461298b116c9c114bb50" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 42015 } }
I20260929 10:28:17.674748 23821 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2619af372df461298b116c9c114bb50 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:28:17.675585 23826 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2619af372df461298b116c9c114bb50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2619af372df461298b116c9c114bb50. Latest consensus state: current_term: 1 leader_uuid: "d2619af372df461298b116c9c114bb50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2619af372df461298b116c9c114bb50" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 42015 } } }
I20260929 10:28:17.675786 23826 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2619af372df461298b116c9c114bb50 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:17.676052 23827 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2619af372df461298b116c9c114bb50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2619af372df461298b116c9c114bb50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2619af372df461298b116c9c114bb50" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 42015 } } }
I20260929 10:28:17.676191 23827 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2619af372df461298b116c9c114bb50 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:17.676301 23835 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:28:17.677525 20848 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:28:17.679770 23835 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:28:17.686714 23835 catalog_manager.cc:1347] Generated new cluster ID: e4d2c158f94f4bdbbdedf164a5d0ede0
I20260929 10:28:17.686870 23835 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:28:17.699821 23824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:17.694383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50986 (local address 127.20.92.62:42015)
0929 10:28:17.694520 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:17.694525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:17.694784 (+ 259us) server_negotiation.cc:407] Connection header received
0929 10:28:17.694972 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:17.694975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:17.695033 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:17.695132 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:17.695705 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:17.696527 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:17.697442 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:17.697729 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:17.697906 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:17.698844 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:17.698848 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:17.698851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:28:17.699324 (+ 473us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:17.699355 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:28:17.699384 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:17.699490 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:17.699571 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 10:28:17.699635 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":57}
I20260929 10:28:17.704296 23835 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:28:17.705247 23835 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:28:17.718442 23835 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2619af372df461298b116c9c114bb50: Generated new TSK 0
I20260929 10:28:17.719244 23835 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:28:17.731146 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 23753
2026-09-29T10:28:17Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1434 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:28:17 dist-test-slave-9cj5 krb5kdc[23854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:28:17 dist-test-slave-9cj5 krb5kdc[23854](info): set up 2 sockets
Sep 29 10:28:17 dist-test-slave-9cj5 krb5kdc[23854](info): commencing operation
krb5kdc: starting...
W20260929 10:28:18.847029 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:28:18 dist-test-slave-9cj5 krb5kdc[23854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:28:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:28:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
I20260929 10:28:18.982642 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:41149
--webserver_interface=127.20.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:41149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:19.098685 23867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:19.099012 23867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:19.099094 23867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:19.103286 23867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:28:19.103379 23867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:19.103415 23867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:19.103451 23867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:28:19.103503 23867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:28:19.109051 23867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:41149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:41149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:19.110281 23867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:19.111286 23867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:19.120378 23873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:19.120358 23872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:19.120358 23875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:19.120661 23867 server_base.cc:1034] running on GCE node
I20260929 10:28:19.121413 23867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:19.122558 23867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:19.123746 23867 hybrid_clock.cc:648] HybridClock initialized: now 1790677699123714 us; error 52 us; skew 500 ppm
Sep 29 10:28:19 dist-test-slave-9cj5 krb5kdc[23854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677699, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:19.126919 23867 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:28:19.128074 23867 webserver.cc:460] Webserver started at http://127.20.92.62:34779/ using document root <none> and password file <none>
I20260929 10:28:19.128733 23867 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:19.128830 23867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:19.129094 23867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:19.131392 23867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "de3f1eba73f943598133482868898803"
format_stamp: "Formatted at 2026-09-29 10:28:19 on dist-test-slave-9cj5"
server_key: "1301fdf1e480e6a5ae4ab6bcbfc14b73"
server_key_iv: "d7d773c49612930d6663900bab8ed217"
server_key_version: "encryptionkey@0"
I20260929 10:28:19.131994 23867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "de3f1eba73f943598133482868898803"
format_stamp: "Formatted at 2026-09-29 10:28:19 on dist-test-slave-9cj5"
server_key: "1301fdf1e480e6a5ae4ab6bcbfc14b73"
server_key_iv: "d7d773c49612930d6663900bab8ed217"
server_key_version: "encryptionkey@0"
I20260929 10:28:19.136181 23867 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:28:19.139047 23882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:19.140463 23867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 10:28:19.140655 23867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "de3f1eba73f943598133482868898803"
format_stamp: "Formatted at 2026-09-29 10:28:19 on dist-test-slave-9cj5"
server_key: "1301fdf1e480e6a5ae4ab6bcbfc14b73"
server_key_iv: "d7d773c49612930d6663900bab8ed217"
server_key_version: "encryptionkey@0"
I20260929 10:28:19.140834 23867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:19.188645 23867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:19.192319 23867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:19.192544 23867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:19.200443 23867 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:41149
I20260929 10:28:19.200752 23934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:41149 every 8 connection(s)
I20260929 10:28:19.201603 23867 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:28:19.204725 23935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:19.210429 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 23867
I20260929 10:28:19.210541 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:28:19.210827 20848 external_mini_cluster.cc:1428] Setting key 392bd7dbceaacc8f84609c9695eb6159
I20260929 10:28:19.211058 23935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de3f1eba73f943598133482868898803: Bootstrap starting.
I20260929 10:28:19.213793 23935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de3f1eba73f943598133482868898803: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:19.214798 23935 log.cc:824] T 00000000000000000000000000000000 P de3f1eba73f943598133482868898803: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:19.216838 23935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de3f1eba73f943598133482868898803: No bootstrap required, opened a new log
Sep 29 10:28:19 dist-test-slave-9cj5 krb5kdc[23854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:19.221097 23935 raft_consensus.cc:348] T 00000000000000000000000000000000 P de3f1eba73f943598133482868898803 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3f1eba73f943598133482868898803" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 41149 } }
I20260929 10:28:19.221336 23935 raft_consensus.cc:374] T 00000000000000000000000000000000 P de3f1eba73f943598133482868898803 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:19.221397 23935 raft_consensus.cc:729] T 00000000000000000000000000000000 P de3f1eba73f943598133482868898803 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de3f1eba73f943598133482868898803, State: Initialized, Role: FOLLOWER
I20260929 10:28:19.221971 23935 consensus_queue.cc:260] T 00000000000000000000000000000000 P de3f1eba73f943598133482868898803 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3f1eba73f943598133482868898803" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 41149 } }
I20260929 10:28:19.222107 23935 raft_consensus.cc:388] T 00000000000000000000000000000000 P de3f1eba73f943598133482868898803 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:28:19.222231 23935 raft_consensus.cc:482] T 00000000000000000000000000000000 P de3f1eba73f943598133482868898803 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:28:19.222359 23935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de3f1eba73f943598133482868898803 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:19.223299 23935 raft_consensus.cc:504] T 00000000000000000000000000000000 P de3f1eba73f943598133482868898803 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3f1eba73f943598133482868898803" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 41149 } }
I20260929 10:28:19.223647 23935 leader_election.cc:302] T 00000000000000000000000000000000 P de3f1eba73f943598133482868898803 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de3f1eba73f943598133482868898803; no voters:
I20260929 10:28:19.224002 23935 leader_election.cc:288] T 00000000000000000000000000000000 P de3f1eba73f943598133482868898803 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:28:19.224360 23940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de3f1eba73f943598133482868898803 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:19.224591 23940 raft_consensus.cc:686] T 00000000000000000000000000000000 P de3f1eba73f943598133482868898803 [term 1 LEADER]: Becoming Leader. State: Replica: de3f1eba73f943598133482868898803, State: Running, Role: LEADER
I20260929 10:28:19.224915 23940 consensus_queue.cc:237] T 00000000000000000000000000000000 P de3f1eba73f943598133482868898803 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3f1eba73f943598133482868898803" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 41149 } }
I20260929 10:28:19.225437 23935 sys_catalog.cc:564] T 00000000000000000000000000000000 P de3f1eba73f943598133482868898803 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:28:19.226711 23942 sys_catalog.cc:455] T 00000000000000000000000000000000 P de3f1eba73f943598133482868898803 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de3f1eba73f943598133482868898803. Latest consensus state: current_term: 1 leader_uuid: "de3f1eba73f943598133482868898803" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3f1eba73f943598133482868898803" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 41149 } } }
I20260929 10:28:19.226866 23942 sys_catalog.cc:458] T 00000000000000000000000000000000 P de3f1eba73f943598133482868898803 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:19.227187 23938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:19.212334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44350 (local address 127.20.92.62:41149)
0929 10:28:19.212807 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:19.212823 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:19.212860 (+ 37us) server_negotiation.cc:407] Connection header received
0929 10:28:19.213518 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:19.213537 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:19.214002 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:19.214316 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:19.215358 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:19.216433 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:19.217479 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:19.218075 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:19.218264 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:19.221218 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:19.221225 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:19.221239 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:19.221268 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:19.223228 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:19.223740 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:19.223750 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:19.223754 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:19.223911 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:19.224226 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:19.224232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:19.224236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:19.224853 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:19.225067 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:19.225697 (+ 630us) server_negotiation.cc:299] Negotiation successful
0929 10:28:19.226072 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":174,"threads_started":1}
I20260929 10:28:19.227263 23941 sys_catalog.cc:455] T 00000000000000000000000000000000 P de3f1eba73f943598133482868898803 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de3f1eba73f943598133482868898803" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3f1eba73f943598133482868898803" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 41149 } } }
I20260929 10:28:19.227347 23941 sys_catalog.cc:458] T 00000000000000000000000000000000 P de3f1eba73f943598133482868898803 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:19.228150 23950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:28:19.231374 23950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:28:19.237089 23950 catalog_manager.cc:1347] Generated new cluster ID: 6808452720ba4e95ab9795953d0cecac
I20260929 10:28:19.237174 23950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:28:19.246546 23950 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:28:19.247896 23950 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:28:19.260775 23950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de3f1eba73f943598133482868898803: Generated new TSK 0
I20260929 10:28:19.261708 23950 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
I20260929 10:28:19.298044 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:0
--local_ip_for_outbound_sockets=127.20.92.1
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:41149
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:19.417528 23961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:19.417932 23961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:19.418025 23961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:19.422168 23961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:19.422286 23961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:19.422395 23961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:28:19.428275 23961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:41149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:19.429524 23961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:19.430599 23961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:19.440922 23966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:19.440986 23969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:19.441015 23961 server_base.cc:1034] running on GCE node
W20260929 10:28:19.441111 23967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:19.441929 23961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:19.442628 23961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:19.443928 23961 hybrid_clock.cc:648] HybridClock initialized: now 1790677699443876 us; error 68 us; skew 500 ppm
Sep 29 10:28:19 dist-test-slave-9cj5 krb5kdc[23854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677699, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:19.447157 23961 init.cc:377] Logged in from keytab as kudu/127.20.92.1@KRBTEST.COM (short username kudu)
I20260929 10:28:19.448128 23961 webserver.cc:460] Webserver started at http://127.20.92.1:39137/ using document root <none> and password file <none>
I20260929 10:28:19.448710 23961 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:19.448802 23961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:19.449046 23961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:19.451279 23961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/data/instance:
uuid: "9cb3da718b224e15ba5bf51e92feb5ba"
format_stamp: "Formatted at 2026-09-29 10:28:19 on dist-test-slave-9cj5"
server_key: "b34c0d59570ef06722f28eab5a0cdb8a"
server_key_iv: "81e320b798bbb2dbe776f1587275672d"
server_key_version: "encryptionkey@0"
I20260929 10:28:19.451874 23961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance:
uuid: "9cb3da718b224e15ba5bf51e92feb5ba"
format_stamp: "Formatted at 2026-09-29 10:28:19 on dist-test-slave-9cj5"
server_key: "b34c0d59570ef06722f28eab5a0cdb8a"
server_key_iv: "81e320b798bbb2dbe776f1587275672d"
server_key_version: "encryptionkey@0"
I20260929 10:28:19.455878 23961 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:28:19.458575 23976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:19.459900 23961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:28:19.460048 23961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "9cb3da718b224e15ba5bf51e92feb5ba"
format_stamp: "Formatted at 2026-09-29 10:28:19 on dist-test-slave-9cj5"
server_key: "b34c0d59570ef06722f28eab5a0cdb8a"
server_key_iv: "81e320b798bbb2dbe776f1587275672d"
server_key_version: "encryptionkey@0"
I20260929 10:28:19.460175 23961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:19.486449 23961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:19.490422 23961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:19.490666 23961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:19.491319 23961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:19.492210 23961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:19.492285 23961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:19.492358 23961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:19.492409 23961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:19.502274 23961 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:38253
I20260929 10:28:19.502291 24089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:38253 every 8 connection(s)
I20260929 10:28:19.503245 23961 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
I20260929 10:28:19.505606 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 23961
I20260929 10:28:19.505753 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance
I20260929 10:28:19.506076 20848 external_mini_cluster.cc:1428] Setting key 996627737d24da4d08d8a4817026f1a0
Sep 29 10:28:19 dist-test-slave-9cj5 krb5kdc[23854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677699, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.92.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.2@KRBTEST.COM" created.
I20260929 10:28:19.520083 23938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:19.505186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:44823 (local address 127.20.92.62:41149)
0929 10:28:19.505343 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:19.505348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:19.506337 (+ 989us) server_negotiation.cc:407] Connection header received
0929 10:28:19.508024 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:19.508028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:19.508082 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:19.508186 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:19.509946 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:19.510485 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:19.511840 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:19.512091 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:19.512262 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:19.515742 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:19.515751 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:19.515754 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:19.515782 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:19.517451 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:19.518167 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:19.518171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:19.518172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:19.518253 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:19.518730 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:19.518733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:19.518735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:19.519004 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:19.519118 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:19.519749 (+ 631us) server_negotiation.cc:299] Negotiation successful
0929 10:28:19.519892 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 10:28:19.520967 24092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:19.505648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41149 (local address 127.20.92.1:44823)
0929 10:28:19.506170 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:19.506223 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:19.507800 (+ 1577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:19.508441 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:19.508450 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:19.508881 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:19.509731 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:19.509791 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:19.510628 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:19.510631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:19.511588 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:19.511598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:19.512263 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:19.512267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:19.512491 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:19.513197 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:19.513226 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:19.515510 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:19.517622 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:19.517633 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:19.517655 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:19.518035 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:19.518374 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:19.518378 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:19.518380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:19.518539 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:19.519130 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:19.519139 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:19.519469 (+ 330us) client_negotiation.cc:769] Sending connection context
0929 10:28:19.519693 (+ 224us) client_negotiation.cc:240] Negotiation successful
0929 10:28:19.520070 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":132,"threads_started":1}
I20260929 10:28:19.522514 24090 heartbeater.cc:347] Connected to a master server at 127.20.92.62:41149
I20260929 10:28:19.522815 24090 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:19.523360 24090 heartbeater.cc:511] Master 127.20.92.62:41149 requested a full tablet report, sending...
I20260929 10:28:19.525254 23899 ts_manager.cc:194] Registered new tserver with Master: 9cb3da718b224e15ba5bf51e92feb5ba (127.20.92.1:38253)
I20260929 10:28:19.527035 23899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.1@KRBTEST.COM'} at 127.20.92.1:44823
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
I20260929 10:28:19.542084 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:0
--local_ip_for_outbound_sockets=127.20.92.2
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:41149
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:19.658818 24095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:19.659117 24095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:19.659163 24095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:19.663398 24095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:19.663488 24095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:19.663553 24095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:28:19.669394 24095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:41149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:19.670533 24095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:19.671519 24095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:19.681949 24101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:19.682269 24095 server_base.cc:1034] running on GCE node
W20260929 10:28:19.681949 24100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:19.682014 24103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:19.682835 24095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:19.683679 24095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:19.684919 24095 hybrid_clock.cc:648] HybridClock initialized: now 1790677699684899 us; error 37 us; skew 500 ppm
Sep 29 10:28:19 dist-test-slave-9cj5 krb5kdc[23854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677699, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:19.688004 24095 init.cc:377] Logged in from keytab as kudu/127.20.92.2@KRBTEST.COM (short username kudu)
I20260929 10:28:19.689101 24095 webserver.cc:460] Webserver started at http://127.20.92.2:35437/ using document root <none> and password file <none>
I20260929 10:28:19.689685 24095 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:19.689819 24095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:19.690059 24095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:19.692407 24095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/data/instance:
uuid: "52882383f9c7406c8592aea002e46047"
format_stamp: "Formatted at 2026-09-29 10:28:19 on dist-test-slave-9cj5"
server_key: "a79e30900694f9e62cdf0a2c9171b00a"
server_key_iv: "1462fcb578aa38bbed0a96f921445f54"
server_key_version: "encryptionkey@0"
I20260929 10:28:19.692996 24095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance:
uuid: "52882383f9c7406c8592aea002e46047"
format_stamp: "Formatted at 2026-09-29 10:28:19 on dist-test-slave-9cj5"
server_key: "a79e30900694f9e62cdf0a2c9171b00a"
server_key_iv: "1462fcb578aa38bbed0a96f921445f54"
server_key_version: "encryptionkey@0"
I20260929 10:28:19.696967 24095 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 10:28:19.699703 24110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:19.701185 24095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:28:19.701347 24095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "52882383f9c7406c8592aea002e46047"
format_stamp: "Formatted at 2026-09-29 10:28:19 on dist-test-slave-9cj5"
server_key: "a79e30900694f9e62cdf0a2c9171b00a"
server_key_iv: "1462fcb578aa38bbed0a96f921445f54"
server_key_version: "encryptionkey@0"
I20260929 10:28:19.701486 24095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:19.736861 24095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:19.740310 24095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:19.740523 24095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:19.741142 24095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:19.742049 24095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:19.742118 24095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:19.742188 24095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:19.742238 24095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:19.751258 24095 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:44523
I20260929 10:28:19.751281 24223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:44523 every 8 connection(s)
I20260929 10:28:19.752152 24095 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
I20260929 10:28:19.759987 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 24095
I20260929 10:28:19.760114 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance
I20260929 10:28:19.760561 20848 external_mini_cluster.cc:1428] Setting key 8db41aba2cbed3cc06f52006bb5b9a20
Sep 29 10:28:19 dist-test-slave-9cj5 krb5kdc[23854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677699, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:19.767158 23938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:19.754098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:52907 (local address 127.20.92.62:41149)
0929 10:28:19.754224 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:19.754228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:19.755096 (+ 868us) server_negotiation.cc:407] Connection header received
0929 10:28:19.756467 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:19.756471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:19.756531 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:19.756611 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:19.757805 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:19.758345 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:19.759487 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:19.759737 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:19.759850 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:19.762753 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:19.762760 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:19.762765 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:19.762796 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:19.764435 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:19.765156 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:19.765162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:19.765167 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:19.765259 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:19.765657 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:19.765663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:19.765668 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:19.765970 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:19.766076 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:19.766793 (+ 717us) server_negotiation.cc:299] Negotiation successful
0929 10:28:19.766945 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 10:28:19.768018 24226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:19.754472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41149 (local address 127.20.92.2:52907)
0929 10:28:19.754937 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:19.754980 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:19.756244 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:19.756780 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:19.756787 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:19.757155 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:19.757632 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:19.757644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:19.758490 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:19.758493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:19.759362 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:19.759371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:19.759897 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:19.759901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:19.760097 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:19.760783 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:19.760798 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:19.762612 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:19.764565 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:19.764574 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:19.764595 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:19.765029 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:19.765382 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:19.765385 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:19.765386 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:19.765550 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:19.766089 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:19.766095 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:19.766399 (+ 304us) client_negotiation.cc:769] Sending connection context
0929 10:28:19.766724 (+ 325us) client_negotiation.cc:240] Negotiation successful
0929 10:28:19.767054 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":102,"threads_started":1}
I20260929 10:28:19.769312 24224 heartbeater.cc:347] Connected to a master server at 127.20.92.62:41149
I20260929 10:28:19.769570 24224 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:19.770126 24224 heartbeater.cc:511] Master 127.20.92.62:41149 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.92.3@KRBTEST.COM; defaulting to no policy
I20260929 10:28:19.771298 23899 ts_manager.cc:194] Registered new tserver with Master: 52882383f9c7406c8592aea002e46047 (127.20.92.2:44523)
I20260929 10:28:19.772017 23899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.2@KRBTEST.COM'} at 127.20.92.2:52907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
I20260929 10:28:19.794593 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:0
--local_ip_for_outbound_sockets=127.20.92.3
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:41149
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:19.921905 24229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:19.922233 24229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:19.922289 24229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:19.927258 24229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:19.927343 24229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:19.927433 24229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:28:19.933687 24229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:41149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:19.934949 24229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:19.935997 24229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:19.946940 24237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:19.946944 24234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:19.946944 24235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:19.947302 24229 server_base.cc:1034] running on GCE node
I20260929 10:28:19.947741 24229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:19.948473 24229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:19.949644 24229 hybrid_clock.cc:648] HybridClock initialized: now 1790677699949607 us; error 54 us; skew 500 ppm
Sep 29 10:28:19 dist-test-slave-9cj5 krb5kdc[23854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677699, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:19.952726 24229 init.cc:377] Logged in from keytab as kudu/127.20.92.3@KRBTEST.COM (short username kudu)
I20260929 10:28:19.953713 24229 webserver.cc:460] Webserver started at http://127.20.92.3:36227/ using document root <none> and password file <none>
I20260929 10:28:19.954398 24229 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:19.954464 24229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:19.954685 24229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:19.956856 24229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/data/instance:
uuid: "354d4a84480f4269b624ef43c5830b1a"
format_stamp: "Formatted at 2026-09-29 10:28:19 on dist-test-slave-9cj5"
server_key: "36658a121decf3a72d58ae1220cb0f75"
server_key_iv: "a366fc80b80a43419aff3b414f7a46f8"
server_key_version: "encryptionkey@0"
I20260929 10:28:19.957465 24229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance:
uuid: "354d4a84480f4269b624ef43c5830b1a"
format_stamp: "Formatted at 2026-09-29 10:28:19 on dist-test-slave-9cj5"
server_key: "36658a121decf3a72d58ae1220cb0f75"
server_key_iv: "a366fc80b80a43419aff3b414f7a46f8"
server_key_version: "encryptionkey@0"
I20260929 10:28:19.961478 24229 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 10:28:19.964519 24244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:19.966199 24229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:28:19.966346 24229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "354d4a84480f4269b624ef43c5830b1a"
format_stamp: "Formatted at 2026-09-29 10:28:19 on dist-test-slave-9cj5"
server_key: "36658a121decf3a72d58ae1220cb0f75"
server_key_iv: "a366fc80b80a43419aff3b414f7a46f8"
server_key_version: "encryptionkey@0"
I20260929 10:28:19.966480 24229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:19.976560 24229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:19.979911 24229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:19.980156 24229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:19.980834 24229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:19.981719 24229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:19.981848 24229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:19.981895 24229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:19.981914 24229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:19.992990 24229 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:38131
I20260929 10:28:19.993111 24357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:38131 every 8 connection(s)
I20260929 10:28:19.994303 24229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:28:20.003614 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 24229
I20260929 10:28:20.003837 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance
I20260929 10:28:20.004177 20848 external_mini_cluster.cc:1428] Setting key 1c4fa03837c6d98d077284380ae1255f
Sep 29 10:28:20 dist-test-slave-9cj5 krb5kdc[23854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677699, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:20.012784 23938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:19.996362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:41103 (local address 127.20.92.62:41149)
0929 10:28:19.996519 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:19.996524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:19.997957 (+ 1433us) server_negotiation.cc:407] Connection header received
0929 10:28:19.999334 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:19.999340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:19.999648 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:19.999856 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:20.001663 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:20.002369 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:20.003857 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:20.004134 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:20.004263 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:20.007548 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:20.007551 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:20.007554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:20.007581 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:20.009242 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:20.010405 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:20.010408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:20.010410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:20.010516 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:20.011208 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:20.011320 (+ 112us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:20.011323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:20.011570 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:20.011690 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:20.012487 (+ 797us) server_negotiation.cc:299] Negotiation successful
0929 10:28:20.012644 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 10:28:20.013652 24360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:19.996633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41149 (local address 127.20.92.3:41103)
0929 10:28:19.997679 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:19.997821 (+ 142us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:19.999018 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:20.000417 (+ 1399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:20.000426 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:20.000822 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:20.001410 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:20.001426 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:20.002537 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:20.002540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:20.003651 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:20.003662 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:20.004379 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:20.004387 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:20.004589 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:20.005466 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:20.005492 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:20.007355 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:20.009467 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:20.009478 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:20.009504 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:20.010194 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:20.010678 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:20.010681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:20.010683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:20.010916 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:20.011786 (+ 870us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:20.011793 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:20.012157 (+ 364us) client_negotiation.cc:769] Sending connection context
0929 10:28:20.012373 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 10:28:20.012683 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":144,"threads_started":1}
I20260929 10:28:20.015249 24358 heartbeater.cc:347] Connected to a master server at 127.20.92.62:41149
I20260929 10:28:20.015547 24358 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:20.016104 24358 heartbeater.cc:511] Master 127.20.92.62:41149 requested a full tablet report, sending...
I20260929 10:28:20.017227 23899 ts_manager.cc:194] Registered new tserver with Master: 354d4a84480f4269b624ef43c5830b1a (127.20.92.3:38131)
I20260929 10:28:20.018172 23899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.3@KRBTEST.COM'} at 127.20.92.3:41103
I20260929 10:28:20.019062 20848 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 10:28:20 dist-test-slave-9cj5 krb5kdc[23854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677700, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 10:28:20 dist-test-slave-9cj5 krb5kdc[23854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677700, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:20.050997 23938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:20.039285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44364 (local address 127.20.92.62:41149)
0929 10:28:20.039472 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:20.039475 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:20.039762 (+ 287us) server_negotiation.cc:407] Connection header received
0929 10:28:20.039841 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:20.039843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:20.039898 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:20.040088 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:20.041028 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:20.041904 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:20.043136 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:20.043403 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:20.043527 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:20.047001 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:20.047005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:20.047007 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:20.047043 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:20.048599 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:20.049295 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:20.049299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:20.049300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:20.049399 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:20.049720 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:20.049723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:20.049724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:20.050120 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:20.050243 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:20.050611 (+ 368us) server_negotiation.cc:299] Negotiation successful
0929 10:28:20.050809 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260929 10:28:20.051671 23899 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44364
I20260929 10:28:20.052904 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 23961
I20260929 10:28:20.060088 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 24095
I20260929 10:28:20.067467 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 24229
I20260929 10:28:20.074447 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 23867
2026-09-29T10:28:20Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2347 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:28:20 dist-test-slave-9cj5 krb5kdc[24370](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:28:20 dist-test-slave-9cj5 krb5kdc[24370](info): set up 2 sockets
Sep 29 10:28:20 dist-test-slave-9cj5 krb5kdc[24370](info): commencing operation
krb5kdc: starting...
W20260929 10:28:21.187875 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.001s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:28:21 dist-test-slave-9cj5 krb5kdc[24370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:28:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:28:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
I20260929 10:28:21.326256 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:44097
--webserver_interface=127.20.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:36985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:44097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:21.444592 24383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:21.444963 24383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:21.445042 24383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:21.449405 24383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:28:21.449539 24383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:21.449579 24383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:21.449616 24383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:28:21.449651 24383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:28:21.455329 24383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:36985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:44097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:44097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:21.456547 24383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:21.457615 24383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:21.466593 24388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:21.466593 24389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:21.466593 24391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:21.466804 24383 server_base.cc:1034] running on GCE node
I20260929 10:28:21.467522 24383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:21.468729 24383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:21.469913 24383 hybrid_clock.cc:648] HybridClock initialized: now 1790677701469873 us; error 52 us; skew 500 ppm
Sep 29 10:28:21 dist-test-slave-9cj5 krb5kdc[24370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677701, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:21.472958 24383 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:28:21.473999 24383 webserver.cc:460] Webserver started at http://127.20.92.62:37407/ using document root <none> and password file <none>
I20260929 10:28:21.474589 24383 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:21.474653 24383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:21.474888 24383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:21.477064 24383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "e625e7c2558d4f77b948a22d4fa3e307"
format_stamp: "Formatted at 2026-09-29 10:28:21 on dist-test-slave-9cj5"
server_key: "928efc070c52e253ad478bb16ce1e7fa"
server_key_iv: "6e7d33dedd953e39f9e6b6fc527c69eb"
server_key_version: "encryptionkey@0"
I20260929 10:28:21.477658 24383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "e625e7c2558d4f77b948a22d4fa3e307"
format_stamp: "Formatted at 2026-09-29 10:28:21 on dist-test-slave-9cj5"
server_key: "928efc070c52e253ad478bb16ce1e7fa"
server_key_iv: "6e7d33dedd953e39f9e6b6fc527c69eb"
server_key_version: "encryptionkey@0"
I20260929 10:28:21.482177 24383 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 10:28:21.486079 24398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:21.487609 24383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:28:21.487795 24383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "e625e7c2558d4f77b948a22d4fa3e307"
format_stamp: "Formatted at 2026-09-29 10:28:21 on dist-test-slave-9cj5"
server_key: "928efc070c52e253ad478bb16ce1e7fa"
server_key_iv: "6e7d33dedd953e39f9e6b6fc527c69eb"
server_key_version: "encryptionkey@0"
I20260929 10:28:21.487947 24383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:21.502117 24383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:21.505960 24383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:21.506196 24383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:21.514285 24383 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:44097
I20260929 10:28:21.514315 24450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:44097 every 8 connection(s)
I20260929 10:28:21.515378 24383 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:28:21.518389 24451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:21.523320 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 24383
I20260929 10:28:21.523478 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:28:21.523762 20848 external_mini_cluster.cc:1428] Setting key b8a4d62d2678c879876da19b46cbcdd0
I20260929 10:28:21.524178 24451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e625e7c2558d4f77b948a22d4fa3e307: Bootstrap starting.
I20260929 10:28:21.526820 24451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e625e7c2558d4f77b948a22d4fa3e307: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:21.527724 24451 log.cc:824] T 00000000000000000000000000000000 P e625e7c2558d4f77b948a22d4fa3e307: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:21.529573 24451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e625e7c2558d4f77b948a22d4fa3e307: No bootstrap required, opened a new log
Sep 29 10:28:21 dist-test-slave-9cj5 krb5kdc[24370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:21.533633 24451 raft_consensus.cc:348] T 00000000000000000000000000000000 P e625e7c2558d4f77b948a22d4fa3e307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e625e7c2558d4f77b948a22d4fa3e307" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 44097 } }
I20260929 10:28:21.533929 24451 raft_consensus.cc:374] T 00000000000000000000000000000000 P e625e7c2558d4f77b948a22d4fa3e307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:21.534022 24451 raft_consensus.cc:729] T 00000000000000000000000000000000 P e625e7c2558d4f77b948a22d4fa3e307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e625e7c2558d4f77b948a22d4fa3e307, State: Initialized, Role: FOLLOWER
I20260929 10:28:21.534559 24451 consensus_queue.cc:260] T 00000000000000000000000000000000 P e625e7c2558d4f77b948a22d4fa3e307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e625e7c2558d4f77b948a22d4fa3e307" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 44097 } }
I20260929 10:28:21.534737 24451 raft_consensus.cc:388] T 00000000000000000000000000000000 P e625e7c2558d4f77b948a22d4fa3e307 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:28:21.534818 24451 raft_consensus.cc:482] T 00000000000000000000000000000000 P e625e7c2558d4f77b948a22d4fa3e307 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:28:21.534920 24451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e625e7c2558d4f77b948a22d4fa3e307 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:21.536144 24451 raft_consensus.cc:504] T 00000000000000000000000000000000 P e625e7c2558d4f77b948a22d4fa3e307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e625e7c2558d4f77b948a22d4fa3e307" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 44097 } }
I20260929 10:28:21.536504 24451 leader_election.cc:302] T 00000000000000000000000000000000 P e625e7c2558d4f77b948a22d4fa3e307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e625e7c2558d4f77b948a22d4fa3e307; no voters:
I20260929 10:28:21.536890 24451 leader_election.cc:288] T 00000000000000000000000000000000 P e625e7c2558d4f77b948a22d4fa3e307 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:28:21.537005 24456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e625e7c2558d4f77b948a22d4fa3e307 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:21.537321 24456 raft_consensus.cc:686] T 00000000000000000000000000000000 P e625e7c2558d4f77b948a22d4fa3e307 [term 1 LEADER]: Becoming Leader. State: Replica: e625e7c2558d4f77b948a22d4fa3e307, State: Running, Role: LEADER
I20260929 10:28:21.537655 24456 consensus_queue.cc:237] T 00000000000000000000000000000000 P e625e7c2558d4f77b948a22d4fa3e307 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e625e7c2558d4f77b948a22d4fa3e307" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 44097 } }
I20260929 10:28:21.538336 24451 sys_catalog.cc:564] T 00000000000000000000000000000000 P e625e7c2558d4f77b948a22d4fa3e307 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:28:21.538935 24456 sys_catalog.cc:455] T 00000000000000000000000000000000 P e625e7c2558d4f77b948a22d4fa3e307 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e625e7c2558d4f77b948a22d4fa3e307. Latest consensus state: current_term: 1 leader_uuid: "e625e7c2558d4f77b948a22d4fa3e307" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e625e7c2558d4f77b948a22d4fa3e307" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 44097 } } }
I20260929 10:28:21.539350 24457 sys_catalog.cc:455] T 00000000000000000000000000000000 P e625e7c2558d4f77b948a22d4fa3e307 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e625e7c2558d4f77b948a22d4fa3e307" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e625e7c2558d4f77b948a22d4fa3e307" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 44097 } } }
I20260929 10:28:21.539443 24457 sys_catalog.cc:458] T 00000000000000000000000000000000 P e625e7c2558d4f77b948a22d4fa3e307 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:21.539861 24454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:21.525238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60336 (local address 127.20.92.62:44097)
0929 10:28:21.525694 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:21.525713 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:21.525785 (+ 72us) server_negotiation.cc:407] Connection header received
0929 10:28:21.526605 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:21.526629 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:21.527070 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:21.527458 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:21.528334 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:21.529316 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:21.530320 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:21.530819 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:21.531000 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:21.533827 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:21.533832 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:21.533846 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:21.533876 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:21.536095 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:21.536611 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:21.536619 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:21.536626 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:21.536767 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:21.537146 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:21.537152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:21.537157 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:21.537669 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:21.537899 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:21.538220 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 10:28:21.538718 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":115,"threads_started":1}
I20260929 10:28:21.539894 24465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:28:21.539150 24456 sys_catalog.cc:458] T 00000000000000000000000000000000 P e625e7c2558d4f77b948a22d4fa3e307 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:21.543150 24465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:28:21.549216 24465 catalog_manager.cc:1347] Generated new cluster ID: ecc398471d7f447ba5db0ff3bc337952
I20260929 10:28:21.549317 24465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:28:21.566504 24465 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:28:21.567963 24465 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:28:21.578820 24465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e625e7c2558d4f77b948a22d4fa3e307: Generated new TSK 0
I20260929 10:28:21.579766 24465 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
I20260929 10:28:21.611191 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:0
--local_ip_for_outbound_sockets=127.20.92.1
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:44097
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:36985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:21.727253 24477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:21.727552 24477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:21.727598 24477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:21.731791 24477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:21.731866 24477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:21.731952 24477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:28:21.737538 24477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:36985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:44097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:21.738718 24477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:21.739857 24477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:21.749954 24483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:21.749954 24482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:21.749970 24485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:21.750257 24477 server_base.cc:1034] running on GCE node
I20260929 10:28:21.750713 24477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:21.751365 24477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:21.752530 24477 hybrid_clock.cc:648] HybridClock initialized: now 1790677701752500 us; error 48 us; skew 500 ppm
Sep 29 10:28:21 dist-test-slave-9cj5 krb5kdc[24370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677701, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:21.755460 24477 init.cc:377] Logged in from keytab as kudu/127.20.92.1@KRBTEST.COM (short username kudu)
I20260929 10:28:21.756449 24477 webserver.cc:460] Webserver started at http://127.20.92.1:43143/ using document root <none> and password file <none>
I20260929 10:28:21.756976 24477 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:21.757040 24477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:21.757215 24477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:21.759546 24477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-0/data/instance:
uuid: "f3643d3778804ee08f697a0e4cfc24ef"
format_stamp: "Formatted at 2026-09-29 10:28:21 on dist-test-slave-9cj5"
server_key: "7536bc8a48a9d1a4b5b79a0b0286d733"
server_key_iv: "5b7701febe0f5b847955686f68af4cd1"
server_key_version: "encryptionkey@0"
I20260929 10:28:21.760107 24477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance:
uuid: "f3643d3778804ee08f697a0e4cfc24ef"
format_stamp: "Formatted at 2026-09-29 10:28:21 on dist-test-slave-9cj5"
server_key: "7536bc8a48a9d1a4b5b79a0b0286d733"
server_key_iv: "5b7701febe0f5b847955686f68af4cd1"
server_key_version: "encryptionkey@0"
I20260929 10:28:21.764011 24477 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:28:21.766607 24492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:21.767915 24477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:28:21.768064 24477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "f3643d3778804ee08f697a0e4cfc24ef"
format_stamp: "Formatted at 2026-09-29 10:28:21 on dist-test-slave-9cj5"
server_key: "7536bc8a48a9d1a4b5b79a0b0286d733"
server_key_iv: "5b7701febe0f5b847955686f68af4cd1"
server_key_version: "encryptionkey@0"
I20260929 10:28:21.768180 24477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:21.787752 24477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:21.791685 24477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:21.791930 24477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:21.792695 24477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:21.793660 24477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:21.793736 24477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:21.793895 24477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:21.793936 24477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:21.804951 24477 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:40011
I20260929 10:28:21.805883 24605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:40011 every 8 connection(s)
I20260929 10:28:21.806175 24477 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
I20260929 10:28:21.808334 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 24477
I20260929 10:28:21.808475 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance
I20260929 10:28:21.808820 20848 external_mini_cluster.cc:1428] Setting key 5f1c96a06283fb8e9f9db02128acfd19
Sep 29 10:28:21 dist-test-slave-9cj5 krb5kdc[24370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677701, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.92.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.2@KRBTEST.COM" created.
I20260929 10:28:21.823983 24454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:21.809682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:38257 (local address 127.20.92.62:44097)
0929 10:28:21.809914 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:21.809922 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:21.810987 (+ 1065us) server_negotiation.cc:407] Connection header received
0929 10:28:21.812560 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:21.812563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:21.812614 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:21.812688 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:21.814266 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:21.815259 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:21.816560 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:21.816898 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:21.817033 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:21.820025 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:21.820028 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:21.820030 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:21.820062 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:21.821624 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:21.822257 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:21.822260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:21.822262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:21.822357 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:21.822731 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:21.822734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:21.822735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:21.822967 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:21.823081 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:21.823650 (+ 569us) server_negotiation.cc:299] Negotiation successful
0929 10:28:21.823785 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260929 10:28:21.824893 24608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:21.810180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:44097 (local address 127.20.92.1:38257)
0929 10:28:21.810825 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:21.810887 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:21.812238 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:21.812888 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:21.812900 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:21.813463 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:21.814103 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:21.814118 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:21.815447 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:21.815450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:21.816425 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:21.816436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:21.817084 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:21.817088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:21.817277 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:21.818009 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:21.818026 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:21.819782 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:21.821790 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:21.821797 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:21.821815 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:21.822136 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:21.822459 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:21.822462 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:21.822464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:21.822623 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:21.823115 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:21.823129 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:21.823395 (+ 266us) client_negotiation.cc:769] Sending connection context
0929 10:28:21.823593 (+ 198us) client_negotiation.cc:240] Negotiation successful
0929 10:28:21.823955 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":175,"threads_started":1}
I20260929 10:28:21.826342 24606 heartbeater.cc:347] Connected to a master server at 127.20.92.62:44097
I20260929 10:28:21.826599 24606 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:21.827108 24606 heartbeater.cc:511] Master 127.20.92.62:44097 requested a full tablet report, sending...
I20260929 10:28:21.829010 24415 ts_manager.cc:194] Registered new tserver with Master: f3643d3778804ee08f697a0e4cfc24ef (127.20.92.1:40011)
I20260929 10:28:21.830726 24415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.1@KRBTEST.COM'} at 127.20.92.1:38257
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
I20260929 10:28:21.842074 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:0
--local_ip_for_outbound_sockets=127.20.92.2
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:44097
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:36985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:21.954995 24611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:21.955271 24611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:21.955317 24611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:21.959587 24611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:21.959666 24611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:21.959728 24611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:28:21.965248 24611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:36985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:44097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:21.966487 24611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:21.967555 24611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:21.977596 24619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:21.977596 24616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:21.978225 24617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:21.978277 24611 server_base.cc:1034] running on GCE node
I20260929 10:28:21.978673 24611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:21.979353 24611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:21.980542 24611 hybrid_clock.cc:648] HybridClock initialized: now 1790677701980523 us; error 31 us; skew 500 ppm
Sep 29 10:28:21 dist-test-slave-9cj5 krb5kdc[24370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677701, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:21.983402 24611 init.cc:377] Logged in from keytab as kudu/127.20.92.2@KRBTEST.COM (short username kudu)
I20260929 10:28:21.984370 24611 webserver.cc:460] Webserver started at http://127.20.92.2:33023/ using document root <none> and password file <none>
I20260929 10:28:21.984900 24611 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:21.984959 24611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:21.985123 24611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:21.987296 24611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-1/data/instance:
uuid: "63cabc9b11744c99aa64ecbddd883a5b"
format_stamp: "Formatted at 2026-09-29 10:28:21 on dist-test-slave-9cj5"
server_key: "a306106a28869f7d346a20ee47fdce76"
server_key_iv: "b050b2769ccb9a06ecadbf6a0f3fdc68"
server_key_version: "encryptionkey@0"
I20260929 10:28:21.987895 24611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance:
uuid: "63cabc9b11744c99aa64ecbddd883a5b"
format_stamp: "Formatted at 2026-09-29 10:28:21 on dist-test-slave-9cj5"
server_key: "a306106a28869f7d346a20ee47fdce76"
server_key_iv: "b050b2769ccb9a06ecadbf6a0f3fdc68"
server_key_version: "encryptionkey@0"
I20260929 10:28:21.992063 24611 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:28:21.995023 24626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:21.996358 24611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:28:21.996490 24611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "63cabc9b11744c99aa64ecbddd883a5b"
format_stamp: "Formatted at 2026-09-29 10:28:21 on dist-test-slave-9cj5"
server_key: "a306106a28869f7d346a20ee47fdce76"
server_key_iv: "b050b2769ccb9a06ecadbf6a0f3fdc68"
server_key_version: "encryptionkey@0"
I20260929 10:28:21.996634 24611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:22.007534 24611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:22.011142 24611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:22.011353 24611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:22.012008 24611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:22.012866 24611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:22.012914 24611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:22.012977 24611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:22.013011 24611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:22.022524 24611 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:38739
I20260929 10:28:22.022549 24739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:38739 every 8 connection(s)
I20260929 10:28:22.023514 24611 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
I20260929 10:28:22.029062 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 24611
I20260929 10:28:22.029249 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance
I20260929 10:28:22.029572 20848 external_mini_cluster.cc:1428] Setting key 892c3a4002acb5571e400ac46dd7e45c
Sep 29 10:28:22 dist-test-slave-9cj5 krb5kdc[24370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677701, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.92.3@KRBTEST.COM; defaulting to no policy
I20260929 10:28:22.041213 24454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:22.025528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:54155 (local address 127.20.92.62:44097)
0929 10:28:22.025675 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:22.025681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:22.026740 (+ 1059us) server_negotiation.cc:407] Connection header received
0929 10:28:22.028379 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:22.028383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:22.028448 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:22.028566 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:22.030140 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:22.030898 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:22.032582 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:22.033011 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:22.033137 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:22.036578 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:22.036581 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:22.036583 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:22.036610 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:22.038325 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:22.039030 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:22.039034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:22.039035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:22.039134 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:22.039546 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:22.039549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:22.039551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:22.039863 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:22.039961 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:22.040888 (+ 927us) server_negotiation.cc:299] Negotiation successful
0929 10:28:22.041021 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 10:28:22.042268 24742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:22.026031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:44097 (local address 127.20.92.2:54155)
0929 10:28:22.026555 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:22.026600 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:22.028129 (+ 1529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:22.028811 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:22.028820 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:22.029274 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:22.029937 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:22.029955 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:22.031211 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:22.031215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:22.032427 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:22.032439 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:22.033250 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:22.033258 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:22.033538 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:22.034329 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:22.034360 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:22.036376 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:22.038532 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:22.038537 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:22.038552 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:22.038913 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:22.039250 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:22.039253 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:22.039256 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:22.039401 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:22.040017 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:22.040029 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:22.040447 (+ 418us) client_negotiation.cc:769] Sending connection context
0929 10:28:22.040754 (+ 307us) client_negotiation.cc:240] Negotiation successful
0929 10:28:22.041190 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":120,"threads_started":1}
I20260929 10:28:22.043699 24740 heartbeater.cc:347] Connected to a master server at 127.20.92.62:44097
I20260929 10:28:22.043960 24740 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:22.044459 24740 heartbeater.cc:511] Master 127.20.92.62:44097 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.3@KRBTEST.COM" created.
I20260929 10:28:22.045719 24415 ts_manager.cc:194] Registered new tserver with Master: 63cabc9b11744c99aa64ecbddd883a5b (127.20.92.2:38739)
I20260929 10:28:22.046581 24415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.2@KRBTEST.COM'} at 127.20.92.2:54155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
I20260929 10:28:22.065968 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:0
--local_ip_for_outbound_sockets=127.20.92.3
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:44097
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:36985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:22.181883 24745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:22.182171 24745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:22.182216 24745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:22.186614 24745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:22.186692 24745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:22.186755 24745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:28:22.192726 24745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:36985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:44097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:22.193950 24745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:22.194943 24745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:22.205096 24753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:22.205179 24745 server_base.cc:1034] running on GCE node
W20260929 10:28:22.205088 24751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:22.205088 24750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:22.205816 24745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:22.206576 24745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:22.207772 24745 hybrid_clock.cc:648] HybridClock initialized: now 1790677702207738 us; error 47 us; skew 500 ppm
Sep 29 10:28:22 dist-test-slave-9cj5 krb5kdc[24370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677702, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:22.210659 24745 init.cc:377] Logged in from keytab as kudu/127.20.92.3@KRBTEST.COM (short username kudu)
I20260929 10:28:22.211617 24745 webserver.cc:460] Webserver started at http://127.20.92.3:35385/ using document root <none> and password file <none>
I20260929 10:28:22.212189 24745 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:22.212262 24745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:22.212432 24745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:22.214538 24745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-2/data/instance:
uuid: "38ff1069af414e208e5819b95ea11139"
format_stamp: "Formatted at 2026-09-29 10:28:22 on dist-test-slave-9cj5"
server_key: "8e37d9eeccec00c5abc2a949729021e7"
server_key_iv: "0cb1728c69b5de2ee8aa3d8a60977283"
server_key_version: "encryptionkey@0"
I20260929 10:28:22.215066 24745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance:
uuid: "38ff1069af414e208e5819b95ea11139"
format_stamp: "Formatted at 2026-09-29 10:28:22 on dist-test-slave-9cj5"
server_key: "8e37d9eeccec00c5abc2a949729021e7"
server_key_iv: "0cb1728c69b5de2ee8aa3d8a60977283"
server_key_version: "encryptionkey@0"
I20260929 10:28:22.219077 24745 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 10:28:22.222184 24760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:22.223809 24745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:28:22.224033 24745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "38ff1069af414e208e5819b95ea11139"
format_stamp: "Formatted at 2026-09-29 10:28:22 on dist-test-slave-9cj5"
server_key: "8e37d9eeccec00c5abc2a949729021e7"
server_key_iv: "0cb1728c69b5de2ee8aa3d8a60977283"
server_key_version: "encryptionkey@0"
I20260929 10:28:22.224170 24745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:22.251783 24745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:22.255523 24745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:22.255692 24745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:22.256349 24745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:22.257205 24745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:22.257251 24745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:22.257290 24745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:22.257305 24745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:22.268009 24745 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:42463
I20260929 10:28:22.268049 24873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:42463 every 8 connection(s)
I20260929 10:28:22.268940 24745 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:28:22.274223 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 24745
I20260929 10:28:22.274410 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance
I20260929 10:28:22.274675 20848 external_mini_cluster.cc:1428] Setting key a41df3c4e6c62aef81e8836358ba0bcd
Sep 29 10:28:22 dist-test-slave-9cj5 krb5kdc[24370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677702, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:22.286823 24454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:22.271111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:39965 (local address 127.20.92.62:44097)
0929 10:28:22.271297 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:22.271301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:22.272154 (+ 853us) server_negotiation.cc:407] Connection header received
0929 10:28:22.274075 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:22.274081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:22.274148 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:22.274251 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:22.276067 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:22.276839 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:22.278452 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:22.278696 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:22.278796 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:22.282567 (+ 3771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:22.282582 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:22.282587 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:22.282621 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:22.284274 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:22.285031 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:22.285034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:22.285036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:22.285140 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:22.285522 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:22.285525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:22.285526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:22.285877 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:22.285973 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:22.286538 (+ 565us) server_negotiation.cc:299] Negotiation successful
0929 10:28:22.286651 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 10:28:22.287839 24876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:22.271503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:44097 (local address 127.20.92.3:39965)
0929 10:28:22.271987 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:22.272030 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:22.273807 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:22.274491 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:22.274502 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:22.275011 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:22.275839 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:22.275869 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:22.276986 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:22.276990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:22.278241 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:22.278251 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:22.278924 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:22.278927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:22.279097 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:22.279798 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:22.279815 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:22.282051 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:22.284462 (+ 2411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:22.284469 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:22.284501 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:22.284883 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:22.285250 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:22.285252 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:22.285254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:22.285406 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:22.285990 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:22.285996 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:22.286267 (+ 271us) client_negotiation.cc:769] Sending connection context
0929 10:28:22.286434 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 10:28:22.286787 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":114,"threads_started":1}
I20260929 10:28:22.289062 24874 heartbeater.cc:347] Connected to a master server at 127.20.92.62:44097
I20260929 10:28:22.289398 24874 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:22.290024 24874 heartbeater.cc:511] Master 127.20.92.62:44097 requested a full tablet report, sending...
I20260929 10:28:22.291149 24415 ts_manager.cc:194] Registered new tserver with Master: 38ff1069af414e208e5819b95ea11139 (127.20.92.3:42463)
I20260929 10:28:22.291749 24415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.3@KRBTEST.COM'} at 127.20.92.3:39965
I20260929 10:28:22.302383 20848 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 10:28:22 dist-test-slave-9cj5 krb5kdc[24370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 10:28:22 dist-test-slave-9cj5 krb5kdc[24370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.1@KRBTEST.COM
I20260929 10:28:22.329895 24881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:22.319348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39642 (local address 127.20.92.1:40011)
0929 10:28:22.319674 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:22.319691 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:22.319719 (+ 28us) server_negotiation.cc:407] Connection header received
0929 10:28:22.319856 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:22.319862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:22.320053 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:22.320225 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:22.320993 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:22.321863 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:22.322761 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:22.323149 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:22.323265 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:22.325947 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:22.325956 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:22.325970 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:22.325995 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:22.327847 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:22.328313 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:22.328318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:22.328319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:22.328483 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:22.328799 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:22.328801 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:22.328802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:22.329064 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:22.329234 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:22.329484 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 10:28:22.329612 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":84,"threads_started":1}
Sep 29 10:28:22 dist-test-slave-9cj5 krb5kdc[24370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:22.341706 24454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:22.332791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51658 (local address 127.20.92.62:44097)
0929 10:28:22.332922 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:22.332925 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:22.332958 (+ 33us) server_negotiation.cc:407] Connection header received
0929 10:28:22.333112 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:22.333114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:22.333159 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:22.333266 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:22.333924 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:22.334443 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:22.335402 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:22.335612 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:22.335694 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:22.338260 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:22.338263 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:22.338265 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:22.338314 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:22.339977 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:22.340406 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:22.340409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:22.340410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:22.340484 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:22.340920 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:22.340923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:22.340924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:22.341151 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:22.341288 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:22.341484 (+ 196us) server_negotiation.cc:299] Negotiation successful
0929 10:28:22.341580 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260929 10:28:22.342178 24415 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51658
I20260929 10:28:22.342826 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 24477
I20260929 10:28:22.348527 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 24611
I20260929 10:28:22.354038 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 24745
I20260929 10:28:22.361155 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 24383
2026-09-29T10:28:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2285 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:28:22 dist-test-slave-9cj5 krb5kdc[24886](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:28:22 dist-test-slave-9cj5 krb5kdc[24886](info): set up 2 sockets
Sep 29 10:28:22 dist-test-slave-9cj5 krb5kdc[24886](info): commencing operation
krb5kdc: starting...
W20260929 10:28:23.423247 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:28:23 dist-test-slave-9cj5 krb5kdc[24886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:28:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:28:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
I20260929 10:28:23.557204 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:45607
--webserver_interface=127.20.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:41943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:45607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:23.679098 24899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:23.679363 24899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:23.679405 24899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:23.683519 24899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:28:23.683578 24899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:23.683597 24899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:23.683614 24899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:28:23.683661 24899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:28:23.689294 24899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:41943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:45607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:45607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:23.690743 24899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:23.691876 24899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:23.700973 24905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:23.700982 24907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:23.700982 24904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:23.701305 24899 server_base.cc:1034] running on GCE node
I20260929 10:28:23.702054 24899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:23.703238 24899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:23.704435 24899 hybrid_clock.cc:648] HybridClock initialized: now 1790677703704421 us; error 28 us; skew 500 ppm
Sep 29 10:28:23 dist-test-slave-9cj5 krb5kdc[24886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677703, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:23.707537 24899 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:28:23.708319 24899 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:23.708395 24899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:23.708648 24899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:23.710858 24899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "f973d459f5a045358aea92f3425d5c80"
format_stamp: "Formatted at 2026-09-29 10:28:23 on dist-test-slave-9cj5"
server_key: "ac04d5bbab9bce6546955b0e2eb96d10"
server_key_iv: "0894ee12ffc0d5a57fc3c9a9abc51927"
server_key_version: "encryptionkey@0"
I20260929 10:28:23.711489 24899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "f973d459f5a045358aea92f3425d5c80"
format_stamp: "Formatted at 2026-09-29 10:28:23 on dist-test-slave-9cj5"
server_key: "ac04d5bbab9bce6546955b0e2eb96d10"
server_key_iv: "0894ee12ffc0d5a57fc3c9a9abc51927"
server_key_version: "encryptionkey@0"
I20260929 10:28:23.715327 24899 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 10:28:23.717942 24913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:23.719331 24899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:28:23.719489 24899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "f973d459f5a045358aea92f3425d5c80"
format_stamp: "Formatted at 2026-09-29 10:28:23 on dist-test-slave-9cj5"
server_key: "ac04d5bbab9bce6546955b0e2eb96d10"
server_key_iv: "0894ee12ffc0d5a57fc3c9a9abc51927"
server_key_version: "encryptionkey@0"
I20260929 10:28:23.719617 24899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:23.729810 24899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:23.733155 24899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:23.733367 24899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:23.740760 24899 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:45607
I20260929 10:28:23.740999 24965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:45607 every 8 connection(s)
I20260929 10:28:23.741441 24899 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:28:23.743994 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 24899
I20260929 10:28:23.744130 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:28:23.744398 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:23.744429 20848 external_mini_cluster.cc:1428] Setting key 862eff9181b1e44f6cbf71240493473a
I20260929 10:28:23.750603 24966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f973d459f5a045358aea92f3425d5c80: Bootstrap starting.
I20260929 10:28:23.752750 24966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f973d459f5a045358aea92f3425d5c80: Neither blocks nor log segments found. Creating new log.
Sep 29 10:28:23 dist-test-slave-9cj5 krb5kdc[24886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:23.753592 24966 log.cc:824] T 00000000000000000000000000000000 P f973d459f5a045358aea92f3425d5c80: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:23.755427 24966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f973d459f5a045358aea92f3425d5c80: No bootstrap required, opened a new log
I20260929 10:28:23.758538 24969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:23.745840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51894 (local address 127.20.92.62:45607)
0929 10:28:23.746276 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:23.746289 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:23.746327 (+ 38us) server_negotiation.cc:407] Connection header received
0929 10:28:23.747147 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:23.747165 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:23.747508 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:23.747916 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:23.748634 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:23.749626 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:23.750523 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:23.751060 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:23.751232 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:23.754008 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:23.754013 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:23.754026 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:23.754053 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:23.755798 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:23.756237 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:23.756247 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:23.756251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:23.756366 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:23.756659 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:23.756662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:23.756664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:23.757042 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:23.757257 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:23.757493 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 10:28:23.757828 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":121,"threads_started":1}
I20260929 10:28:23.759032 24966 raft_consensus.cc:348] T 00000000000000000000000000000000 P f973d459f5a045358aea92f3425d5c80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f973d459f5a045358aea92f3425d5c80" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45607 } }
I20260929 10:28:23.759290 24966 raft_consensus.cc:374] T 00000000000000000000000000000000 P f973d459f5a045358aea92f3425d5c80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:23.759338 24966 raft_consensus.cc:729] T 00000000000000000000000000000000 P f973d459f5a045358aea92f3425d5c80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f973d459f5a045358aea92f3425d5c80, State: Initialized, Role: FOLLOWER
I20260929 10:28:23.759765 24966 consensus_queue.cc:260] T 00000000000000000000000000000000 P f973d459f5a045358aea92f3425d5c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f973d459f5a045358aea92f3425d5c80" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45607 } }
I20260929 10:28:23.759887 24966 raft_consensus.cc:388] T 00000000000000000000000000000000 P f973d459f5a045358aea92f3425d5c80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:28:23.759967 24966 raft_consensus.cc:482] T 00000000000000000000000000000000 P f973d459f5a045358aea92f3425d5c80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:28:23.760067 24966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f973d459f5a045358aea92f3425d5c80 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:23.761085 24966 raft_consensus.cc:504] T 00000000000000000000000000000000 P f973d459f5a045358aea92f3425d5c80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f973d459f5a045358aea92f3425d5c80" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45607 } }
I20260929 10:28:23.761428 24966 leader_election.cc:302] T 00000000000000000000000000000000 P f973d459f5a045358aea92f3425d5c80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f973d459f5a045358aea92f3425d5c80; no voters:
I20260929 10:28:23.761729 24966 leader_election.cc:288] T 00000000000000000000000000000000 P f973d459f5a045358aea92f3425d5c80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:28:23.761938 24971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f973d459f5a045358aea92f3425d5c80 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:23.762194 24971 raft_consensus.cc:686] T 00000000000000000000000000000000 P f973d459f5a045358aea92f3425d5c80 [term 1 LEADER]: Becoming Leader. State: Replica: f973d459f5a045358aea92f3425d5c80, State: Running, Role: LEADER
I20260929 10:28:23.762526 24971 consensus_queue.cc:237] T 00000000000000000000000000000000 P f973d459f5a045358aea92f3425d5c80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f973d459f5a045358aea92f3425d5c80" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45607 } }
I20260929 10:28:23.762990 24966 sys_catalog.cc:564] T 00000000000000000000000000000000 P f973d459f5a045358aea92f3425d5c80 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:28:23.764623 24972 sys_catalog.cc:455] T 00000000000000000000000000000000 P f973d459f5a045358aea92f3425d5c80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f973d459f5a045358aea92f3425d5c80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f973d459f5a045358aea92f3425d5c80" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45607 } } }
I20260929 10:28:23.764626 24973 sys_catalog.cc:455] T 00000000000000000000000000000000 P f973d459f5a045358aea92f3425d5c80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f973d459f5a045358aea92f3425d5c80. Latest consensus state: current_term: 1 leader_uuid: "f973d459f5a045358aea92f3425d5c80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f973d459f5a045358aea92f3425d5c80" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45607 } } }
I20260929 10:28:23.764850 24972 sys_catalog.cc:458] T 00000000000000000000000000000000 P f973d459f5a045358aea92f3425d5c80 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:23.764850 24973 sys_catalog.cc:458] T 00000000000000000000000000000000 P f973d459f5a045358aea92f3425d5c80 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:23.765422 24983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:28:23.768740 24983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:28:23.775480 24983 catalog_manager.cc:1347] Generated new cluster ID: e95bc79180e24f8cbb938d23dc9a23c0
I20260929 10:28:23.775583 24983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:28:23.793479 24983 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:28:23.794556 24983 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:28:23.802302 24983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f973d459f5a045358aea92f3425d5c80: Generated new TSK 0
I20260929 10:28:23.803381 24983 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
I20260929 10:28:23.841912 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:0
--local_ip_for_outbound_sockets=127.20.92.1
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:45607
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:41943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:23.972558 24992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:23.972847 24992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:23.972894 24992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:23.977231 24992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:23.977313 24992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:23.977411 24992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:28:23.983031 24992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:41943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:45607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:23.984176 24992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:23.985486 24992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:23.996008 24997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:23.996018 25000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:23.996007 24998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:23.996074 24992 server_base.cc:1034] running on GCE node
I20260929 10:28:23.996809 24992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:23.997510 24992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:23.998711 24992 hybrid_clock.cc:648] HybridClock initialized: now 1790677703998679 us; error 50 us; skew 500 ppm
Sep 29 10:28:23 dist-test-slave-9cj5 krb5kdc[24886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677703, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:24.001623 24992 init.cc:377] Logged in from keytab as kudu/127.20.92.1@KRBTEST.COM (short username kudu)
I20260929 10:28:24.002581 24992 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:24.002683 24992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:24.002911 24992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:24.005121 24992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-0/data/instance:
uuid: "9d85fb03fe804cb89857f93b21f71b51"
format_stamp: "Formatted at 2026-09-29 10:28:23 on dist-test-slave-9cj5"
server_key: "d112d2e1faa59f16cf96c5cc433f9c09"
server_key_iv: "88f8439fb6dde8132306bc8146a06703"
server_key_version: "encryptionkey@0"
I20260929 10:28:24.005712 24992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance:
uuid: "9d85fb03fe804cb89857f93b21f71b51"
format_stamp: "Formatted at 2026-09-29 10:28:23 on dist-test-slave-9cj5"
server_key: "d112d2e1faa59f16cf96c5cc433f9c09"
server_key_iv: "88f8439fb6dde8132306bc8146a06703"
server_key_version: "encryptionkey@0"
I20260929 10:28:24.009814 24992 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 10:28:24.012526 25006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:24.013893 24992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:28:24.014024 24992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "9d85fb03fe804cb89857f93b21f71b51"
format_stamp: "Formatted at 2026-09-29 10:28:23 on dist-test-slave-9cj5"
server_key: "d112d2e1faa59f16cf96c5cc433f9c09"
server_key_iv: "88f8439fb6dde8132306bc8146a06703"
server_key_version: "encryptionkey@0"
I20260929 10:28:24.014133 24992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:24.029469 24992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:24.032536 24992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:24.032749 24992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:24.033320 24992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:24.034221 24992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:24.034289 24992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:24.034359 24992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:24.034401 24992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:24.043404 24992 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:40981
I20260929 10:28:24.043478 25119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:40981 every 8 connection(s)
I20260929 10:28:24.044085 24992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
I20260929 10:28:24.049484 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 24992
I20260929 10:28:24.049625 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance
I20260929 10:28:24.049938 20848 external_mini_cluster.cc:1428] Setting key fb38f8cbd08fb53ce5bcefe66915b623
Sep 29 10:28:24 dist-test-slave-9cj5 krb5kdc[24886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677703, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:24.058473 24969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:24.045899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:46291 (local address 127.20.92.62:45607)
0929 10:28:24.046033 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:24.046036 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:24.046873 (+ 837us) server_negotiation.cc:407] Connection header received
0929 10:28:24.048156 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:24.048159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:24.048218 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:24.048325 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:24.049464 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.050197 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.051562 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.051801 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.051917 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:24.054670 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:24.054674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:24.054676 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:24.054699 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:24.056305 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:24.056857 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:24.056861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:24.056862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:24.056946 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:24.057297 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:24.057302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:24.057304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:24.057489 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:24.057598 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:24.058169 (+ 571us) server_negotiation.cc:299] Negotiation successful
0929 10:28:24.058300 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.20.92.2@KRBTEST.COM; defaulting to no policy
I20260929 10:28:24.059342 25122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:24.046272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45607 (local address 127.20.92.1:46291)
0929 10:28:24.046710 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:24.046763 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:24.047935 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:24.048499 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:24.048506 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:24.048856 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:24.049318 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:24.049331 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.050323 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.050326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:24.051440 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:24.051450 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.051950 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.051954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:24.052121 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:24.052747 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:24.052762 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:24.054472 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:24.056454 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:24.056460 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:24.056482 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:24.056740 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:24.057050 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:24.057055 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:24.057061 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:24.057191 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:24.057596 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:24.057603 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:24.057923 (+ 320us) client_negotiation.cc:769] Sending connection context
0929 10:28:24.058116 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 10:28:24.058450 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":109,"threads_started":1}
I20260929 10:28:24.060606 25120 heartbeater.cc:347] Connected to a master server at 127.20.92.62:45607
I20260929 10:28:24.061026 25120 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:24.061551 25120 heartbeater.cc:511] Master 127.20.92.62:45607 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.2@KRBTEST.COM" created.
I20260929 10:28:24.063409 24930 ts_manager.cc:194] Registered new tserver with Master: 9d85fb03fe804cb89857f93b21f71b51 (127.20.92.1:40981)
I20260929 10:28:24.064774 24930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.1@KRBTEST.COM'} at 127.20.92.1:46291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
I20260929 10:28:24.082190 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:0
--local_ip_for_outbound_sockets=127.20.92.2
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:45607
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:41943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:24.194702 25125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:24.194975 25125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:24.195022 25125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:24.199090 25125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:24.199149 25125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:24.199213 25125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:28:24.204536 25125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:41943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:45607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:24.205622 25125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:24.206727 25125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:24.216300 25130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:24.216364 25131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:24.216300 25133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:24.216646 25125 server_base.cc:1034] running on GCE node
I20260929 10:28:24.217211 25125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:24.217967 25125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:24.219151 25125 hybrid_clock.cc:648] HybridClock initialized: now 1790677704219129 us; error 42 us; skew 500 ppm
Sep 29 10:28:24 dist-test-slave-9cj5 krb5kdc[24886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677704, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:24.225394 25125 init.cc:377] Logged in from keytab as kudu/127.20.92.2@KRBTEST.COM (short username kudu)
I20260929 10:28:24.226294 25125 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:24.226366 25125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:24.226545 25125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:24.228736 25125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-1/data/instance:
uuid: "69a49b21ddbb48e48d49c39f5ac35193"
format_stamp: "Formatted at 2026-09-29 10:28:24 on dist-test-slave-9cj5"
server_key: "7948bffbb6bed80fe483aa61796d14c7"
server_key_iv: "ba2bb9a7b93949cb9b925a75957b26bb"
server_key_version: "encryptionkey@0"
I20260929 10:28:24.229315 25125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance:
uuid: "69a49b21ddbb48e48d49c39f5ac35193"
format_stamp: "Formatted at 2026-09-29 10:28:24 on dist-test-slave-9cj5"
server_key: "7948bffbb6bed80fe483aa61796d14c7"
server_key_iv: "ba2bb9a7b93949cb9b925a75957b26bb"
server_key_version: "encryptionkey@0"
I20260929 10:28:24.233311 25125 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 10:28:24.236095 25139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:24.237510 25125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:28:24.237668 25125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "69a49b21ddbb48e48d49c39f5ac35193"
format_stamp: "Formatted at 2026-09-29 10:28:24 on dist-test-slave-9cj5"
server_key: "7948bffbb6bed80fe483aa61796d14c7"
server_key_iv: "ba2bb9a7b93949cb9b925a75957b26bb"
server_key_version: "encryptionkey@0"
I20260929 10:28:24.237843 25125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:24.267673 25125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:24.271183 25125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:24.271431 25125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:24.272060 25125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:24.272923 25125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:24.272971 25125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:24.273012 25125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:24.273072 25125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:24.282195 25125 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:36721
I20260929 10:28:24.282217 25252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:36721 every 8 connection(s)
I20260929 10:28:24.282871 25125 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
I20260929 10:28:24.289124 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 25125
I20260929 10:28:24.289222 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance
I20260929 10:28:24.289527 20848 external_mini_cluster.cc:1428] Setting key 536295d19c94f225cea9804b53473eed
Sep 29 10:28:24 dist-test-slave-9cj5 krb5kdc[24886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677704, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:24.299417 24969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:24.284795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:53401 (local address 127.20.92.62:45607)
0929 10:28:24.284977 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:24.284981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:24.286473 (+ 1492us) server_negotiation.cc:407] Connection header received
0929 10:28:24.287358 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:24.287361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:24.287433 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:24.287518 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:24.288852 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.289470 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.291144 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.291498 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.291658 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:24.294809 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:24.294816 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:24.294822 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:24.294859 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:24.296667 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:24.297480 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:24.297483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:24.297484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:24.297564 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:24.298102 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:24.298105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:24.298106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:24.298329 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:24.298411 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:24.299147 (+ 736us) server_negotiation.cc:299] Negotiation successful
0929 10:28:24.299287 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
WARNING: no policy specified for kudu/127.20.92.3@KRBTEST.COM; defaulting to no policy
I20260929 10:28:24.300662 25255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:24.285219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45607 (local address 127.20.92.2:53401)
0929 10:28:24.286293 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:24.286351 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:24.287122 (+ 771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:24.287727 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:24.287734 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:24.288116 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:24.288606 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:24.288620 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.289634 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.289637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:24.290786 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:24.290806 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.291715 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.291718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:24.291927 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:24.292625 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:24.292644 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:24.294599 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:24.296819 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:24.296829 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:24.296851 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:24.297314 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:24.297668 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:24.297673 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:24.297677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:24.297951 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:24.298448 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:24.298457 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:24.298821 (+ 364us) client_negotiation.cc:769] Sending connection context
0929 10:28:24.299068 (+ 247us) client_negotiation.cc:240] Negotiation successful
0929 10:28:24.299473 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":123,"threads_started":1}
I20260929 10:28:24.302124 25253 heartbeater.cc:347] Connected to a master server at 127.20.92.62:45607
I20260929 10:28:24.302354 25253 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:24.302938 25253 heartbeater.cc:511] Master 127.20.92.62:45607 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.3@KRBTEST.COM" created.
I20260929 10:28:24.304133 24930 ts_manager.cc:194] Registered new tserver with Master: 69a49b21ddbb48e48d49c39f5ac35193 (127.20.92.2:36721)
I20260929 10:28:24.305003 24930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.2@KRBTEST.COM'} at 127.20.92.2:53401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
I20260929 10:28:24.321105 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:0
--local_ip_for_outbound_sockets=127.20.92.3
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:45607
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:41943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:24.436411 25258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:24.436738 25258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:24.436830 25258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:24.441074 25258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:24.441170 25258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:24.441296 25258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:28:24.447041 25258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:41943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:45607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:24.448168 25258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:24.449236 25258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:28:24.459054 25258 server_base.cc:1034] running on GCE node
W20260929 10:28:24.459061 25266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:24.459052 25263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:24.459075 25264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:24.459728 25258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:24.460407 25258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:24.461597 25258 hybrid_clock.cc:648] HybridClock initialized: now 1790677704461533 us; error 77 us; skew 500 ppm
Sep 29 10:28:24 dist-test-slave-9cj5 krb5kdc[24886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677704, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:24.464556 25258 init.cc:377] Logged in from keytab as kudu/127.20.92.3@KRBTEST.COM (short username kudu)
I20260929 10:28:24.465348 25258 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:24.465442 25258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:24.465670 25258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:24.467794 25258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-2/data/instance:
uuid: "febcdfe204f34d11b4aef9be12bfacb7"
format_stamp: "Formatted at 2026-09-29 10:28:24 on dist-test-slave-9cj5"
server_key: "4541f14233eaca89a09077642792e4f6"
server_key_iv: "ce7932fbc916736e0b44be6f4f0e5de7"
server_key_version: "encryptionkey@0"
I20260929 10:28:24.468384 25258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance:
uuid: "febcdfe204f34d11b4aef9be12bfacb7"
format_stamp: "Formatted at 2026-09-29 10:28:24 on dist-test-slave-9cj5"
server_key: "4541f14233eaca89a09077642792e4f6"
server_key_iv: "ce7932fbc916736e0b44be6f4f0e5de7"
server_key_version: "encryptionkey@0"
I20260929 10:28:24.472146 25258 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260929 10:28:24.474797 25272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:24.476119 25258 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:28:24.476267 25258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "febcdfe204f34d11b4aef9be12bfacb7"
format_stamp: "Formatted at 2026-09-29 10:28:24 on dist-test-slave-9cj5"
server_key: "4541f14233eaca89a09077642792e4f6"
server_key_iv: "ce7932fbc916736e0b44be6f4f0e5de7"
server_key_version: "encryptionkey@0"
I20260929 10:28:24.476404 25258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:24.496129 25258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:24.499754 25258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:24.499967 25258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:24.500550 25258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:24.501400 25258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:24.501471 25258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:24.501542 25258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:24.501595 25258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:24.511212 25258 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:45927
I20260929 10:28:24.511228 25385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:45927 every 8 connection(s)
I20260929 10:28:24.511848 25258 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:28:24.518175 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 25258
I20260929 10:28:24.518357 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance
I20260929 10:28:24.518719 20848 external_mini_cluster.cc:1428] Setting key 6f6bdb6819c0e0a38aba5d4e0db8cedc
Sep 29 10:28:24 dist-test-slave-9cj5 krb5kdc[24886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677704, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:24.527217 24969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:24.513953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:36681 (local address 127.20.92.62:45607)
0929 10:28:24.514128 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:24.514132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:24.515049 (+ 917us) server_negotiation.cc:407] Connection header received
0929 10:28:24.516356 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:24.516360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:24.516419 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:24.516522 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:24.517793 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.518463 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.519838 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.520061 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.520174 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:24.523091 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:24.523095 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:24.523099 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:24.523137 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:24.524730 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:24.525331 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:24.525337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:24.525340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:24.525440 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:24.525909 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:24.525915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:24.525917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:24.526142 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:24.526268 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:24.526847 (+ 579us) server_negotiation.cc:299] Negotiation successful
0929 10:28:24.526992 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 10:28:24.528110 25388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:24.514390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45607 (local address 127.20.92.3:36681)
0929 10:28:24.514888 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:24.514932 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:24.516120 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:24.516694 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:24.516701 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:24.517130 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:24.517606 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:24.517619 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.518591 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.518594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:24.519695 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:24.519705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.520340 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.520343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:24.520478 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:24.521329 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:24.521356 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:24.522876 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:24.524870 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:24.524878 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:24.524896 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:24.525207 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:24.525553 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:24.525559 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:24.525561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:24.525730 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:24.526277 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:24.526287 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:24.526561 (+ 274us) client_negotiation.cc:769] Sending connection context
0929 10:28:24.526780 (+ 219us) client_negotiation.cc:240] Negotiation successful
0929 10:28:24.527141 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":120,"threads_started":1}
I20260929 10:28:24.529361 25386 heartbeater.cc:347] Connected to a master server at 127.20.92.62:45607
I20260929 10:28:24.529597 25386 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:24.530192 25386 heartbeater.cc:511] Master 127.20.92.62:45607 requested a full tablet report, sending...
I20260929 10:28:24.531208 24930 ts_manager.cc:194] Registered new tserver with Master: febcdfe204f34d11b4aef9be12bfacb7 (127.20.92.3:45927)
I20260929 10:28:24.531824 24930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.3@KRBTEST.COM'} at 127.20.92.3:36681
I20260929 10:28:24.533517 20848 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:28:24.544062 24969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:24.536123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51896 (local address 127.20.92.62:45607)
0929 10:28:24.536287 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:24.536294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:24.536516 (+ 222us) server_negotiation.cc:407] Connection header received
0929 10:28:24.536588 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:24.536590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:24.536630 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:24.536710 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:24.537396 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.538035 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.538884 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.539090 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.539172 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:24.540725 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:24.540727 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:24.540730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:24.540752 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:24.542257 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:24.542740 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:24.542745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:24.542747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:24.542830 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:24.543116 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:24.543119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:24.543121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:24.543375 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:24.543519 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:24.543759 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 10:28:24.543891 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 10:28:24.547664 24929 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:28:24.550699 24929 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:28:24.565778 25399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:24.559379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47690 (local address 127.20.92.2:36721)
0929 10:28:24.560782 (+ 1403us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:24.560792 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:24.560811 (+ 19us) server_negotiation.cc:407] Connection header received
0929 10:28:24.560872 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:24.560875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:24.561096 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:24.561252 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:24.561540 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.562564 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.564136 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.564965 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.565080 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:24.565234 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:24.565503 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 10:28:24.565604 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1301,"thread_start_us":112,"threads_started":1}
I20260929 10:28:24.565771 25397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:24.559252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:36721 (local address 127.0.0.1:47690)
0929 10:28:24.559912 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:24.559962 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:24.560085 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:24.561249 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:24.561252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:24.561269 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:24.561404 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:24.561411 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.562699 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.562702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:24.564000 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:24.564011 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.565086 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.565089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:24.565314 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:24.565341 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 10:28:24.565460 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 10:28:24.565580 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":110,"thread_start_us":87,"threads_started":1}
I20260929 10:28:24.565850 25402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:24.560036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35636 (local address 127.20.92.3:45927)
0929 10:28:24.560615 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:24.560626 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:24.560645 (+ 19us) server_negotiation.cc:407] Connection header received
0929 10:28:24.560732 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:24.560737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:24.560944 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:24.561145 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:24.561482 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.562391 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.564151 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.564761 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.564883 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:24.565041 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:24.565561 (+ 520us) server_negotiation.cc:299] Negotiation successful
0929 10:28:24.565665 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":152,"threads_started":1}
I20260929 10:28:24.566213 25400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:24.559562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:45927 (local address 127.0.0.1:35636)
0929 10:28:24.560452 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:24.560462 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:24.560524 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:24.561141 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:24.561152 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:24.561179 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:24.561350 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:24.561358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.562520 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.562523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:24.563982 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:24.563991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.565104 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.565107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:24.565300 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:24.565340 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 10:28:24.565443 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 10:28:24.566050 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"mutex_wait_us":590,"thread_start_us":104,"threads_started":1}
I20260929 10:28:24.566449 25401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:24.560047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37612 (local address 127.20.92.1:40981)
0929 10:28:24.561441 (+ 1394us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:24.561450 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:24.561473 (+ 23us) server_negotiation.cc:407] Connection header received
0929 10:28:24.561546 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:24.561550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:24.561721 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:24.561881 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:24.562240 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.563382 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.564808 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.565622 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.565730 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:24.565900 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:24.566200 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 10:28:24.566300 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1297,"thread_start_us":123,"threads_started":1}
I20260929 10:28:24.566946 25398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:24.559290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:40981 (local address 127.0.0.1:37612)
0929 10:28:24.560112 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:24.560126 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:24.560223 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:24.562014 (+ 1791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:24.562020 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:24.562036 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:24.562117 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:24.562123 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.563509 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.563512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:24.564648 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:24.564658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.565965 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.565970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:24.566062 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:24.566082 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 10:28:24.566154 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 10:28:24.566210 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"thread_start_us":81,"threads_started":1}
I20260929 10:28:24.568873 25320 tablet_service.cc:1467] Processing CreateTablet for tablet 6059ed36b96a4787b82999535d968396 (DEFAULT_TABLE table=test-table [id=77e409cefa4f4d4c9ad26374233392ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:24.568872 25187 tablet_service.cc:1467] Processing CreateTablet for tablet 6059ed36b96a4787b82999535d968396 (DEFAULT_TABLE table=test-table [id=77e409cefa4f4d4c9ad26374233392ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:24.569824 25187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6059ed36b96a4787b82999535d968396. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:24.569842 25320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6059ed36b96a4787b82999535d968396. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:24.576079 25403 tablet_bootstrap.cc:492] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193: Bootstrap starting.
I20260929 10:28:24.576197 25404 tablet_bootstrap.cc:492] T 6059ed36b96a4787b82999535d968396 P febcdfe204f34d11b4aef9be12bfacb7: Bootstrap starting.
I20260929 10:28:24.578147 25404 tablet_bootstrap.cc:654] T 6059ed36b96a4787b82999535d968396 P febcdfe204f34d11b4aef9be12bfacb7: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:24.578598 25403 tablet_bootstrap.cc:654] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:24.578186 25054 tablet_service.cc:1467] Processing CreateTablet for tablet 6059ed36b96a4787b82999535d968396 (DEFAULT_TABLE table=test-table [id=77e409cefa4f4d4c9ad26374233392ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:24.579315 25054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6059ed36b96a4787b82999535d968396. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:24.579317 25404 log.cc:824] T 6059ed36b96a4787b82999535d968396 P febcdfe204f34d11b4aef9be12bfacb7: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:24.579375 25403 log.cc:824] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:24.581306 25404 tablet_bootstrap.cc:492] T 6059ed36b96a4787b82999535d968396 P febcdfe204f34d11b4aef9be12bfacb7: No bootstrap required, opened a new log
I20260929 10:28:24.581306 25403 tablet_bootstrap.cc:492] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193: No bootstrap required, opened a new log
I20260929 10:28:24.581498 25403 ts_tablet_manager.cc:1397] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 10:28:24.581532 25404 ts_tablet_manager.cc:1397] T 6059ed36b96a4787b82999535d968396 P febcdfe204f34d11b4aef9be12bfacb7: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 10:28:24.585531 25407 tablet_bootstrap.cc:492] T 6059ed36b96a4787b82999535d968396 P 9d85fb03fe804cb89857f93b21f71b51: Bootstrap starting.
I20260929 10:28:24.585467 25403 raft_consensus.cc:348] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a49b21ddbb48e48d49c39f5ac35193" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 36721 } } peers { permanent_uuid: "9d85fb03fe804cb89857f93b21f71b51" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 40981 } } peers { permanent_uuid: "febcdfe204f34d11b4aef9be12bfacb7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 45927 } }
I20260929 10:28:24.585467 25404 raft_consensus.cc:348] T 6059ed36b96a4787b82999535d968396 P febcdfe204f34d11b4aef9be12bfacb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a49b21ddbb48e48d49c39f5ac35193" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 36721 } } peers { permanent_uuid: "9d85fb03fe804cb89857f93b21f71b51" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 40981 } } peers { permanent_uuid: "febcdfe204f34d11b4aef9be12bfacb7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 45927 } }
I20260929 10:28:24.585783 25404 raft_consensus.cc:374] T 6059ed36b96a4787b82999535d968396 P febcdfe204f34d11b4aef9be12bfacb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:24.585857 25404 raft_consensus.cc:729] T 6059ed36b96a4787b82999535d968396 P febcdfe204f34d11b4aef9be12bfacb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: febcdfe204f34d11b4aef9be12bfacb7, State: Initialized, Role: FOLLOWER
I20260929 10:28:24.585929 25403 raft_consensus.cc:374] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:24.585990 25403 raft_consensus.cc:729] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69a49b21ddbb48e48d49c39f5ac35193, State: Initialized, Role: FOLLOWER
I20260929 10:28:24.586499 25403 consensus_queue.cc:260] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a49b21ddbb48e48d49c39f5ac35193" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 36721 } } peers { permanent_uuid: "9d85fb03fe804cb89857f93b21f71b51" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 40981 } } peers { permanent_uuid: "febcdfe204f34d11b4aef9be12bfacb7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 45927 } }
I20260929 10:28:24.586735 25404 consensus_queue.cc:260] T 6059ed36b96a4787b82999535d968396 P febcdfe204f34d11b4aef9be12bfacb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a49b21ddbb48e48d49c39f5ac35193" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 36721 } } peers { permanent_uuid: "9d85fb03fe804cb89857f93b21f71b51" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 40981 } } peers { permanent_uuid: "febcdfe204f34d11b4aef9be12bfacb7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 45927 } }
I20260929 10:28:24.587217 25253 heartbeater.cc:503] Master 127.20.92.62:45607 was elected leader, sending a full tablet report...
I20260929 10:28:24.587580 25403 ts_tablet_manager.cc:1428] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 10:28:24.588033 25407 tablet_bootstrap.cc:654] T 6059ed36b96a4787b82999535d968396 P 9d85fb03fe804cb89857f93b21f71b51: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:24.588896 25386 heartbeater.cc:503] Master 127.20.92.62:45607 was elected leader, sending a full tablet report...
I20260929 10:28:24.588946 25407 log.cc:824] T 6059ed36b96a4787b82999535d968396 P 9d85fb03fe804cb89857f93b21f71b51: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:24.589241 25404 ts_tablet_manager.cc:1428] T 6059ed36b96a4787b82999535d968396 P febcdfe204f34d11b4aef9be12bfacb7: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260929 10:28:24.591252 25407 tablet_bootstrap.cc:492] T 6059ed36b96a4787b82999535d968396 P 9d85fb03fe804cb89857f93b21f71b51: No bootstrap required, opened a new log
I20260929 10:28:24.591473 25407 ts_tablet_manager.cc:1397] T 6059ed36b96a4787b82999535d968396 P 9d85fb03fe804cb89857f93b21f71b51: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 10:28:24.595082 25407 raft_consensus.cc:348] T 6059ed36b96a4787b82999535d968396 P 9d85fb03fe804cb89857f93b21f71b51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a49b21ddbb48e48d49c39f5ac35193" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 36721 } } peers { permanent_uuid: "9d85fb03fe804cb89857f93b21f71b51" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 40981 } } peers { permanent_uuid: "febcdfe204f34d11b4aef9be12bfacb7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 45927 } }
I20260929 10:28:24.595314 25407 raft_consensus.cc:374] T 6059ed36b96a4787b82999535d968396 P 9d85fb03fe804cb89857f93b21f71b51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:24.595348 25407 raft_consensus.cc:729] T 6059ed36b96a4787b82999535d968396 P 9d85fb03fe804cb89857f93b21f71b51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d85fb03fe804cb89857f93b21f71b51, State: Initialized, Role: FOLLOWER
I20260929 10:28:24.595711 25407 consensus_queue.cc:260] T 6059ed36b96a4787b82999535d968396 P 9d85fb03fe804cb89857f93b21f71b51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a49b21ddbb48e48d49c39f5ac35193" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 36721 } } peers { permanent_uuid: "9d85fb03fe804cb89857f93b21f71b51" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 40981 } } peers { permanent_uuid: "febcdfe204f34d11b4aef9be12bfacb7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 45927 } }
I20260929 10:28:24.596385 25120 heartbeater.cc:503] Master 127.20.92.62:45607 was elected leader, sending a full tablet report...
I20260929 10:28:24.596597 25407 ts_tablet_manager.cc:1428] T 6059ed36b96a4787b82999535d968396 P 9d85fb03fe804cb89857f93b21f71b51: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 10:28:24.630803 25408 raft_consensus.cc:482] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:28:24.631059 25408 raft_consensus.cc:504] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a49b21ddbb48e48d49c39f5ac35193" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 36721 } } peers { permanent_uuid: "9d85fb03fe804cb89857f93b21f71b51" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 40981 } } peers { permanent_uuid: "febcdfe204f34d11b4aef9be12bfacb7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 45927 } }
I20260929 10:28:24.632298 25408 leader_election.cc:288] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d85fb03fe804cb89857f93b21f71b51 (127.20.92.1:40981), febcdfe204f34d11b4aef9be12bfacb7 (127.20.92.3:45927)
I20260929 10:28:24.636941 25255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:24.632586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:45927 (local address 127.20.92.2:59877)
0929 10:28:24.632696 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:24.632714 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:24.632816 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:24.633110 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:24.633112 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:24.633126 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:24.633280 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:24.633285 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.634514 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.634517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:24.635689 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:24.635710 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.636501 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.636505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:24.636623 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:24.636644 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 10:28:24.636719 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 10:28:24.636791 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":69}
I20260929 10:28:24.636989 25402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:24.632682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:59877 (local address 127.20.92.3:45927)
0929 10:28:24.632854 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:24.632860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:24.632879 (+ 19us) server_negotiation.cc:407] Connection header received
0929 10:28:24.632934 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:24.632940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:24.632996 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:24.633073 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:24.633438 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.634261 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.635832 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.636379 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.636466 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:24.636520 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:24.636749 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 10:28:24.636828 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 10:28:24.637120 25412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:24.632585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:40981 (local address 127.20.92.2:39751)
0929 10:28:24.632999 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:24.633009 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:24.633120 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:24.633359 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:24.633363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:24.633383 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:24.633477 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:24.633481 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.634450 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.634453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:24.635739 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:24.635749 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.636639 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.636642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:24.636817 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:24.636838 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 10:28:24.636918 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 10:28:24.636977 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":247,"thread_start_us":123,"threads_started":1}
I20260929 10:28:24.637174 25401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:24.632685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:39751 (local address 127.20.92.1:40981)
0929 10:28:24.632810 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:24.632815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:24.633121 (+ 306us) server_negotiation.cc:407] Connection header received
0929 10:28:24.633203 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:24.633206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:24.633258 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:24.633318 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:24.633580 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.634355 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.635945 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.636503 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.636610 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:24.636674 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:24.636952 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 10:28:24.637040 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 10:28:24.637893 25340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6059ed36b96a4787b82999535d968396" candidate_uuid: "69a49b21ddbb48e48d49c39f5ac35193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "febcdfe204f34d11b4aef9be12bfacb7" is_pre_election: true
I20260929 10:28:24.638062 25074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6059ed36b96a4787b82999535d968396" candidate_uuid: "69a49b21ddbb48e48d49c39f5ac35193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d85fb03fe804cb89857f93b21f71b51" is_pre_election: true
I20260929 10:28:24.638231 25340 raft_consensus.cc:2454] T 6059ed36b96a4787b82999535d968396 P febcdfe204f34d11b4aef9be12bfacb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69a49b21ddbb48e48d49c39f5ac35193 in term 0.
I20260929 10:28:24.638309 25074 raft_consensus.cc:2454] T 6059ed36b96a4787b82999535d968396 P 9d85fb03fe804cb89857f93b21f71b51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69a49b21ddbb48e48d49c39f5ac35193 in term 0.
I20260929 10:28:24.638818 25141 leader_election.cc:302] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69a49b21ddbb48e48d49c39f5ac35193, 9d85fb03fe804cb89857f93b21f71b51; no voters:
I20260929 10:28:24.639144 25408 raft_consensus.cc:2781] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:28:24.639253 25408 raft_consensus.cc:482] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:28:24.639314 25408 raft_consensus.cc:3037] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:24.640422 25408 raft_consensus.cc:504] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a49b21ddbb48e48d49c39f5ac35193" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 36721 } } peers { permanent_uuid: "9d85fb03fe804cb89857f93b21f71b51" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 40981 } } peers { permanent_uuid: "febcdfe204f34d11b4aef9be12bfacb7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 45927 } }
I20260929 10:28:24.640815 25408 leader_election.cc:288] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193 [CANDIDATE]: Term 1 election: Requested vote from peers 9d85fb03fe804cb89857f93b21f71b51 (127.20.92.1:40981), febcdfe204f34d11b4aef9be12bfacb7 (127.20.92.3:45927)
I20260929 10:28:24.641211 25340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6059ed36b96a4787b82999535d968396" candidate_uuid: "69a49b21ddbb48e48d49c39f5ac35193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "febcdfe204f34d11b4aef9be12bfacb7"
I20260929 10:28:24.641258 25074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6059ed36b96a4787b82999535d968396" candidate_uuid: "69a49b21ddbb48e48d49c39f5ac35193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d85fb03fe804cb89857f93b21f71b51"
I20260929 10:28:24.641350 25340 raft_consensus.cc:3037] T 6059ed36b96a4787b82999535d968396 P febcdfe204f34d11b4aef9be12bfacb7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:24.641387 25074 raft_consensus.cc:3037] T 6059ed36b96a4787b82999535d968396 P 9d85fb03fe804cb89857f93b21f71b51 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:24.642601 25074 raft_consensus.cc:2454] T 6059ed36b96a4787b82999535d968396 P 9d85fb03fe804cb89857f93b21f71b51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69a49b21ddbb48e48d49c39f5ac35193 in term 1.
I20260929 10:28:24.642607 25340 raft_consensus.cc:2454] T 6059ed36b96a4787b82999535d968396 P febcdfe204f34d11b4aef9be12bfacb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69a49b21ddbb48e48d49c39f5ac35193 in term 1.
I20260929 10:28:24.642951 25141 leader_election.cc:302] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69a49b21ddbb48e48d49c39f5ac35193, 9d85fb03fe804cb89857f93b21f71b51; no voters:
I20260929 10:28:24.643282 25408 raft_consensus.cc:2781] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:24.643522 25408 raft_consensus.cc:686] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193 [term 1 LEADER]: Becoming Leader. State: Replica: 69a49b21ddbb48e48d49c39f5ac35193, State: Running, Role: LEADER
I20260929 10:28:24.643874 25408 consensus_queue.cc:237] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a49b21ddbb48e48d49c39f5ac35193" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 36721 } } peers { permanent_uuid: "9d85fb03fe804cb89857f93b21f71b51" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 40981 } } peers { permanent_uuid: "febcdfe204f34d11b4aef9be12bfacb7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 45927 } }
I20260929 10:28:24.647045 24929 catalog_manager.cc:5495] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69a49b21ddbb48e48d49c39f5ac35193 (127.20.92.2). New cstate: current_term: 1 leader_uuid: "69a49b21ddbb48e48d49c39f5ac35193" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a49b21ddbb48e48d49c39f5ac35193" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 36721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d85fb03fe804cb89857f93b21f71b51" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 40981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "febcdfe204f34d11b4aef9be12bfacb7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 45927 } health_report { overall_health: UNKNOWN } } }
I20260929 10:28:24.664417 25399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:24.660473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47698 (local address 127.20.92.2:36721)
0929 10:28:24.660647 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:24.660651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:24.660667 (+ 16us) server_negotiation.cc:407] Connection header received
0929 10:28:24.660827 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:24.660829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:24.660897 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:24.660994 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:28:24.661311 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.661890 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.662838 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:24.663045 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:24.663126 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:24.663448 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:28:24.663889 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:28:24.664005 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:24.664131 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 10:28:24.664183 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 10:28:24.670452 25074 raft_consensus.cc:1261] T 6059ed36b96a4787b82999535d968396 P 9d85fb03fe804cb89857f93b21f71b51 [term 1 FOLLOWER]: Refusing update from remote peer 69a49b21ddbb48e48d49c39f5ac35193: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:24.670452 25340 raft_consensus.cc:1261] T 6059ed36b96a4787b82999535d968396 P febcdfe204f34d11b4aef9be12bfacb7 [term 1 FOLLOWER]: Refusing update from remote peer 69a49b21ddbb48e48d49c39f5ac35193: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:24.671154 25408 consensus_queue.cc:1035] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d85fb03fe804cb89857f93b21f71b51" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 40981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:28:24.671382 25413 consensus_queue.cc:1035] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193 [LEADER]: Connected to new peer: Peer: permanent_uuid: "febcdfe204f34d11b4aef9be12bfacb7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 45927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:28:24.679523 25418 mvcc.cc:205] Tried to move back new op lower bound from 7334615878322688000 to 7334615878223888384. Current Snapshot: MvccSnapshot[applied={T|T < 7334615878322688000}]
I20260929 10:28:24.680683 25417 mvcc.cc:205] Tried to move back new op lower bound from 7334615878322688000 to 7334615878223888384. Current Snapshot: MvccSnapshot[applied={T|T < 7334615878322688000}]
I20260929 10:28:24.687636 24928 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:28:24.687829 24928 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:28:24.690519 24928 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f973d459f5a045358aea92f3425d5c80: Sending DeleteTablet for 3 replicas of tablet 6059ed36b96a4787b82999535d968396
I20260929 10:28:24.691253 25187 tablet_service.cc:1514] Processing DeleteTablet for tablet 6059ed36b96a4787b82999535d968396 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:24 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:47690
I20260929 10:28:24.691327 25054 tablet_service.cc:1514] Processing DeleteTablet for tablet 6059ed36b96a4787b82999535d968396 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:24 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:37612
I20260929 10:28:24.691442 25320 tablet_service.cc:1514] Processing DeleteTablet for tablet 6059ed36b96a4787b82999535d968396 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:24 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:35636
I20260929 10:28:24.691808 25427 tablet_replica.cc:330] T 6059ed36b96a4787b82999535d968396 P 9d85fb03fe804cb89857f93b21f71b51: stopping tablet replica
I20260929 10:28:24.691920 25426 tablet_replica.cc:330] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193: stopping tablet replica
I20260929 10:28:24.692183 25427 raft_consensus.cc:2229] T 6059ed36b96a4787b82999535d968396 P 9d85fb03fe804cb89857f93b21f71b51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:28:24.692242 25426 raft_consensus.cc:2229] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:28:24.692591 25427 raft_consensus.cc:2258] T 6059ed36b96a4787b82999535d968396 P 9d85fb03fe804cb89857f93b21f71b51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:24.692621 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 24992
I20260929 10:28:24.692682 25426 raft_consensus.cc:2258] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:24.694669 25426 ts_tablet_manager.cc:1905] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:24.698304 25426 ts_tablet_manager.cc:1918] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:28:24.698462 25426 log.cc:1196] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-1/wal/wals/6059ed36b96a4787b82999535d968396
I20260929 10:28:24.698873 25426 ts_tablet_manager.cc:1939] T 6059ed36b96a4787b82999535d968396 P 69a49b21ddbb48e48d49c39f5ac35193: Deleting consensus metadata
I20260929 10:28:24.699510 25428 tablet_replica.cc:330] T 6059ed36b96a4787b82999535d968396 P febcdfe204f34d11b4aef9be12bfacb7: stopping tablet replica
I20260929 10:28:24.699909 25428 raft_consensus.cc:2229] T 6059ed36b96a4787b82999535d968396 P febcdfe204f34d11b4aef9be12bfacb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:28:24.700178 24917 catalog_manager.cc:4841] TS 69a49b21ddbb48e48d49c39f5ac35193 (127.20.92.2:36721): tablet 6059ed36b96a4787b82999535d968396 (table test-table [id=77e409cefa4f4d4c9ad26374233392ed]) successfully deleted
I20260929 10:28:24.700327 25428 raft_consensus.cc:2258] T 6059ed36b96a4787b82999535d968396 P febcdfe204f34d11b4aef9be12bfacb7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 10:28:24.701099 24915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.92.1:40981 (error 108)
I20260929 10:28:24.701462 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 25125
I20260929 10:28:24.702793 25398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:24.702460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:40981 (local address 127.0.0.1:37616)
0929 10:28:24.702606 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:24.702688 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.1:40981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 10:28:24.703058 25428 ts_tablet_manager.cc:1905] T 6059ed36b96a4787b82999535d968396 P febcdfe204f34d11b4aef9be12bfacb7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 10:28:24.703269 24915 catalog_manager.cc:4584] TS 9d85fb03fe804cb89857f93b21f71b51 (127.20.92.1:40981): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6059ed36b96a4787b82999535d968396: Network error: Client connection negotiation failed: client connection to 127.20.92.1:40981: connect: Connection refused (error 111)
I20260929 10:28:24.705312 25428 ts_tablet_manager.cc:1918] T 6059ed36b96a4787b82999535d968396 P febcdfe204f34d11b4aef9be12bfacb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:28:24.705394 25428 log.cc:1196] T 6059ed36b96a4787b82999535d968396 P febcdfe204f34d11b4aef9be12bfacb7: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677679287822-20848-0/minicluster-data/ts-2/wal/wals/6059ed36b96a4787b82999535d968396
I20260929 10:28:24.705806 25428 ts_tablet_manager.cc:1939] T 6059ed36b96a4787b82999535d968396 P febcdfe204f34d11b4aef9be12bfacb7: Deleting consensus metadata
I20260929 10:28:24.706976 24914 catalog_manager.cc:4841] TS febcdfe204f34d11b4aef9be12bfacb7 (127.20.92.3:45927): tablet 6059ed36b96a4787b82999535d968396 (table test-table [id=77e409cefa4f4d4c9ad26374233392ed]) successfully deleted
I20260929 10:28:24.708042 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 25258
I20260929 10:28:24.713681 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 24899
2026-09-29T10:28:24Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2351 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T10:28:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:28:24Z Disabled control of system clock
I20260929 10:28:24.744369 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:34017
--webserver_interface=127.20.92.62
--webserver_port=0
--builtin_ntp_servers=127.20.92.20:38743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:34017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 10:28:24.866093 25433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:24.866393 25433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:24.866433 25433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:24.870685 25433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:28:24.870757 25433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:24.870774 25433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:24.870791 25433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:28:24.870807 25433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:28:24.875772 25433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:38743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:34017
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:34017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:24.876878 25433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:24.878314 25433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:24.887939 25438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:24.888011 25439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:24.888077 25433 server_base.cc:1034] running on GCE node
W20260929 10:28:24.888017 25441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:24.888964 25433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:24.890244 25433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:24.891428 25433 hybrid_clock.cc:648] HybridClock initialized: now 1790677704891395 us; error 50 us; skew 500 ppm
I20260929 10:28:24.893679 25433 webserver.cc:460] Webserver started at http://127.20.92.62:34985/ using document root <none> and password file <none>
I20260929 10:28:24.894322 25433 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:24.894393 25433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:24.894567 25433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:24.897011 25433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "9ac70a4553bb45e7a7045e3d72d8396f"
format_stamp: "Formatted at 2026-09-29 10:28:24 on dist-test-slave-9cj5"
server_key: "27a7482d2b7904b3135d7e6e5e33f8ab"
server_key_iv: "538963d92f7d5e65481e8d6c057a85a9"
server_key_version: "encryptionkey@0"
I20260929 10:28:24.897619 25433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "9ac70a4553bb45e7a7045e3d72d8396f"
format_stamp: "Formatted at 2026-09-29 10:28:24 on dist-test-slave-9cj5"
server_key: "27a7482d2b7904b3135d7e6e5e33f8ab"
server_key_iv: "538963d92f7d5e65481e8d6c057a85a9"
server_key_version: "encryptionkey@0"
I20260929 10:28:24.903085 25433 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 10:28:24.906226 25447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:24.907716 25433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 10:28:24.907948 25433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "9ac70a4553bb45e7a7045e3d72d8396f"
format_stamp: "Formatted at 2026-09-29 10:28:24 on dist-test-slave-9cj5"
server_key: "27a7482d2b7904b3135d7e6e5e33f8ab"
server_key_iv: "538963d92f7d5e65481e8d6c057a85a9"
server_key_version: "encryptionkey@0"
I20260929 10:28:24.908092 25433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:24.913002 25433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:24.913638 25433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:24.913892 25433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:24.921298 25433 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:34017
I20260929 10:28:24.921316 25499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:34017 every 8 connection(s)
I20260929 10:28:24.922307 25433 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:28:24.925532 25500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:24.930989 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 25433
I20260929 10:28:24.931105 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:28:24.931421 20848 external_mini_cluster.cc:1428] Setting key 0d8d620701532e99397754447419d281
I20260929 10:28:24.931411 25500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ac70a4553bb45e7a7045e3d72d8396f: Bootstrap starting.
I20260929 10:28:24.934113 25500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ac70a4553bb45e7a7045e3d72d8396f: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:24.934851 25500 log.cc:824] T 00000000000000000000000000000000 P 9ac70a4553bb45e7a7045e3d72d8396f: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:24.937095 25500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ac70a4553bb45e7a7045e3d72d8396f: No bootstrap required, opened a new log
I20260929 10:28:24.937286 25503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:24.932961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42680 (local address 127.20.92.62:34017)
0929 10:28:24.933610 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:24.933624 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:24.933658 (+ 34us) server_negotiation.cc:407] Connection header received
0929 10:28:24.934483 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:24.934501 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:24.934859 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:24.935181 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:24.936006 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:24.936010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:24.936021 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:28:24.936031 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:24.936082 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:28:24.936097 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:24.936183 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:24.936312 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 10:28:24.936564 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":131,"threads_started":1}
I20260929 10:28:24.940940 25500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ac70a4553bb45e7a7045e3d72d8396f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac70a4553bb45e7a7045e3d72d8396f" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34017 } }
I20260929 10:28:24.941186 25500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ac70a4553bb45e7a7045e3d72d8396f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:24.941247 25500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ac70a4553bb45e7a7045e3d72d8396f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ac70a4553bb45e7a7045e3d72d8396f, State: Initialized, Role: FOLLOWER
I20260929 10:28:24.941651 25500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ac70a4553bb45e7a7045e3d72d8396f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac70a4553bb45e7a7045e3d72d8396f" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34017 } }
I20260929 10:28:24.941859 25500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ac70a4553bb45e7a7045e3d72d8396f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:28:24.941954 25500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ac70a4553bb45e7a7045e3d72d8396f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:28:24.942044 25500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ac70a4553bb45e7a7045e3d72d8396f [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:24.943081 25500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ac70a4553bb45e7a7045e3d72d8396f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac70a4553bb45e7a7045e3d72d8396f" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34017 } }
I20260929 10:28:24.943375 25500 leader_election.cc:302] T 00000000000000000000000000000000 P 9ac70a4553bb45e7a7045e3d72d8396f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ac70a4553bb45e7a7045e3d72d8396f; no voters:
I20260929 10:28:24.943629 25500 leader_election.cc:288] T 00000000000000000000000000000000 P 9ac70a4553bb45e7a7045e3d72d8396f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:28:24.943742 25505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ac70a4553bb45e7a7045e3d72d8396f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:24.943990 25505 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ac70a4553bb45e7a7045e3d72d8396f [term 1 LEADER]: Becoming Leader. State: Replica: 9ac70a4553bb45e7a7045e3d72d8396f, State: Running, Role: LEADER
I20260929 10:28:24.944334 25505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ac70a4553bb45e7a7045e3d72d8396f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac70a4553bb45e7a7045e3d72d8396f" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34017 } }
I20260929 10:28:24.944927 25500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ac70a4553bb45e7a7045e3d72d8396f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:28:24.946043 25507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ac70a4553bb45e7a7045e3d72d8396f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ac70a4553bb45e7a7045e3d72d8396f. Latest consensus state: current_term: 1 leader_uuid: "9ac70a4553bb45e7a7045e3d72d8396f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac70a4553bb45e7a7045e3d72d8396f" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34017 } } }
I20260929 10:28:24.946055 25506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ac70a4553bb45e7a7045e3d72d8396f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ac70a4553bb45e7a7045e3d72d8396f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac70a4553bb45e7a7045e3d72d8396f" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34017 } } }
I20260929 10:28:24.946193 25507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ac70a4553bb45e7a7045e3d72d8396f [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:24.946200 25506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ac70a4553bb45e7a7045e3d72d8396f [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:24.946584 25514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:28:24.949796 25514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:28:24.954936 25514 catalog_manager.cc:1347] Generated new cluster ID: 32561c47d4cc4b3abbe45e05ef8a1805
I20260929 10:28:24.955008 25514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:28:24.966639 25514 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:28:24.967006 25514 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:28:24.980681 25514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ac70a4553bb45e7a7045e3d72d8396f: Generated new TSK 0
I20260929 10:28:24.981606 25514 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:28:24.985498 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:0
--local_ip_for_outbound_sockets=127.20.92.1
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:34017
--builtin_ntp_servers=127.20.92.20:38743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 10:28:25.110854 25524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:25.111217 25524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:25.111265 25524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:25.117168 25524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:25.117260 25524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:25.117326 25524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:28:25.122594 25524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:38743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:34017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:25.123818 25524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:25.125339 25524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:25.138332 25530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:25.138350 25529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:25.138484 25524 server_base.cc:1034] running on GCE node
W20260929 10:28:25.138357 25532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:25.139248 25524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:25.139976 25524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:25.141208 25524 hybrid_clock.cc:648] HybridClock initialized: now 1790677705141185 us; error 32 us; skew 500 ppm
I20260929 10:28:25.143751 25524 webserver.cc:460] Webserver started at http://127.20.92.1:37571/ using document root <none> and password file <none>
I20260929 10:28:25.144425 25524 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:25.144498 25524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:25.144790 25524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:25.147433 25524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-0/data/instance:
uuid: "b5cf22164fcc4c658bd421338d86904f"
format_stamp: "Formatted at 2026-09-29 10:28:25 on dist-test-slave-9cj5"
server_key: "11b2260c0d6eb61497d6c381ca22eeaa"
server_key_iv: "ba25c6d38a286441ef4ee56bba48e16e"
server_key_version: "encryptionkey@0"
I20260929 10:28:25.148061 25524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance:
uuid: "b5cf22164fcc4c658bd421338d86904f"
format_stamp: "Formatted at 2026-09-29 10:28:25 on dist-test-slave-9cj5"
server_key: "11b2260c0d6eb61497d6c381ca22eeaa"
server_key_iv: "ba25c6d38a286441ef4ee56bba48e16e"
server_key_version: "encryptionkey@0"
I20260929 10:28:25.152344 25524 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:28:25.155774 25538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:25.157730 25524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:28:25.157979 25524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "b5cf22164fcc4c658bd421338d86904f"
format_stamp: "Formatted at 2026-09-29 10:28:25 on dist-test-slave-9cj5"
server_key: "11b2260c0d6eb61497d6c381ca22eeaa"
server_key_iv: "ba25c6d38a286441ef4ee56bba48e16e"
server_key_version: "encryptionkey@0"
I20260929 10:28:25.158114 25524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:25.163213 25524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:25.163865 25524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:25.164053 25524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:25.164608 25524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:25.165565 25524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:25.165637 25524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:25.165699 25524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:25.165746 25524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:25.176857 25524 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:40189
I20260929 10:28:25.176927 25651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:40189 every 8 connection(s)
I20260929 10:28:25.178053 25524 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
I20260929 10:28:25.183172 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 25524
I20260929 10:28:25.183493 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance
I20260929 10:28:25.183931 20848 external_mini_cluster.cc:1428] Setting key 3b980c2627449c3ebdfce9abe008c480
I20260929 10:28:25.187997 25503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:25.180577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:36289 (local address 127.20.92.62:34017)
0929 10:28:25.180747 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:25.180751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:25.182244 (+ 1493us) server_negotiation.cc:407] Connection header received
0929 10:28:25.184515 (+ 2271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:25.184520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:25.184578 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:25.184692 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:25.187133 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:25.187139 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:25.187146 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:28:25.187170 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:25.187198 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:28:25.187208 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:25.187311 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:25.187663 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 10:28:25.187754 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 10:28:25.188045 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:0
--local_ip_for_outbound_sockets=127.20.92.2
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:34017
--builtin_ntp_servers=127.20.92.20:38743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 10:28:25.190243 25654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:25.181003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:34017 (local address 127.20.92.1:36289)
0929 10:28:25.182070 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:25.182116 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:25.183918 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:25.185479 (+ 1561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:25.185488 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:25.185598 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:25.186537 (+ 939us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:28:25.186539 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:25.186600 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:28:25.186700 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:25.187339 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:25.187359 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:25.187366 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 10:28:25.187626 (+ 260us) client_negotiation.cc:240] Negotiation successful
0929 10:28:25.188232 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":134,"threads_started":1}
I20260929 10:28:25.192668 25652 heartbeater.cc:347] Connected to a master server at 127.20.92.62:34017
I20260929 10:28:25.193425 25652 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:25.194293 25652 heartbeater.cc:511] Master 127.20.92.62:34017 requested a full tablet report, sending...
I20260929 10:28:25.196960 25464 ts_manager.cc:194] Registered new tserver with Master: b5cf22164fcc4c658bd421338d86904f (127.20.92.1:40189)
W20260929 10:28:25.325860 25655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:25.326314 25655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:25.326413 25655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:25.330758 25655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:25.330826 25655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:25.330885 25655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:28:25.335853 25655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:38743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:34017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:25.336988 25655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:25.338446 25655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:25.348944 25663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:25.348991 25660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:25.348968 25661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:25.349200 25655 server_base.cc:1034] running on GCE node
I20260929 10:28:25.349658 25655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:25.350399 25655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:25.351570 25655 hybrid_clock.cc:648] HybridClock initialized: now 1790677705351555 us; error 28 us; skew 500 ppm
I20260929 10:28:25.353612 25655 webserver.cc:460] Webserver started at http://127.20.92.2:39747/ using document root <none> and password file <none>
I20260929 10:28:25.354262 25655 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:25.354329 25655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:25.354545 25655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:25.356659 25655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-1/data/instance:
uuid: "2e97dc86141d4c4790841c02662ec2f1"
format_stamp: "Formatted at 2026-09-29 10:28:25 on dist-test-slave-9cj5"
server_key: "b27812a7984b485d30c394690e95862f"
server_key_iv: "d24771f43dc4bb25bacddabb1aa450b5"
server_key_version: "encryptionkey@0"
I20260929 10:28:25.357275 25655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance:
uuid: "2e97dc86141d4c4790841c02662ec2f1"
format_stamp: "Formatted at 2026-09-29 10:28:25 on dist-test-slave-9cj5"
server_key: "b27812a7984b485d30c394690e95862f"
server_key_iv: "d24771f43dc4bb25bacddabb1aa450b5"
server_key_version: "encryptionkey@0"
I20260929 10:28:25.361614 25655 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 10:28:25.364607 25669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:25.366178 25655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:28:25.366330 25655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "2e97dc86141d4c4790841c02662ec2f1"
format_stamp: "Formatted at 2026-09-29 10:28:25 on dist-test-slave-9cj5"
server_key: "b27812a7984b485d30c394690e95862f"
server_key_iv: "d24771f43dc4bb25bacddabb1aa450b5"
server_key_version: "encryptionkey@0"
I20260929 10:28:25.366462 25655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:25.371467 25655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:25.372097 25655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:25.372287 25655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:25.372876 25655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:25.374058 25655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:25.374132 25655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:25.374202 25655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:25.374254 25655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:25.383970 25655 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:33897
I20260929 10:28:25.384047 25782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:33897 every 8 connection(s)
I20260929 10:28:25.384934 25655 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
I20260929 10:28:25.387970 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 25655
I20260929 10:28:25.388098 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance
I20260929 10:28:25.388412 20848 external_mini_cluster.cc:1428] Setting key 9852388db26162771ae9be4324bfac05
I20260929 10:28:25.390590 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:0
--local_ip_for_outbound_sockets=127.20.92.3
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:34017
--builtin_ntp_servers=127.20.92.20:38743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 10:28:25.392860 25503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:25.386953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:58425 (local address 127.20.92.62:34017)
0929 10:28:25.387121 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:25.387124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:25.388481 (+ 1357us) server_negotiation.cc:407] Connection header received
0929 10:28:25.390176 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:25.390180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:25.390219 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:25.390308 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:25.391940 (+ 1632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:25.391947 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:25.391953 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:28:25.391974 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:25.392007 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:28:25.392022 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:25.392200 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:25.392323 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 10:28:25.392404 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 10:28:25.393887 25785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:25.387351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:34017 (local address 127.20.92.2:58425)
0929 10:28:25.388308 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:25.388365 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:25.389952 (+ 1587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:25.390477 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:25.390485 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:25.390582 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:25.391593 (+ 1011us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:28:25.391594 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:25.391640 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:28:25.391715 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:25.392146 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:25.392153 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:25.392165 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 10:28:25.392384 (+ 219us) client_negotiation.cc:240] Negotiation successful
0929 10:28:25.392693 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":115,"threads_started":1}
I20260929 10:28:25.395249 25783 heartbeater.cc:347] Connected to a master server at 127.20.92.62:34017
I20260929 10:28:25.395522 25783 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:25.396034 25783 heartbeater.cc:511] Master 127.20.92.62:34017 requested a full tablet report, sending...
I20260929 10:28:25.397199 25464 ts_manager.cc:194] Registered new tserver with Master: 2e97dc86141d4c4790841c02662ec2f1 (127.20.92.2:33897)
W20260929 10:28:25.508008 25786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:25.508335 25786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:25.508426 25786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:25.512686 25786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:25.512773 25786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:25.512861 25786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:28:25.517814 25786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:38743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:34017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:25.518967 25786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:25.520372 25786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:28:25.531143 25786 server_base.cc:1034] running on GCE node
W20260929 10:28:25.531113 25791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:25.531083 25792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:25.531072 25794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:25.531859 25786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:25.532572 25786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:25.533824 25786 hybrid_clock.cc:648] HybridClock initialized: now 1790677705533802 us; error 68 us; skew 500 ppm
I20260929 10:28:25.535812 25786 webserver.cc:460] Webserver started at http://127.20.92.3:43139/ using document root <none> and password file <none>
I20260929 10:28:25.536422 25786 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:25.536518 25786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:25.536751 25786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:25.538971 25786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-2/data/instance:
uuid: "8d52fd55ae0d483cae0ea5723d7ed696"
format_stamp: "Formatted at 2026-09-29 10:28:25 on dist-test-slave-9cj5"
server_key: "6b144fac7a8089fe8b98d41af99c2c1e"
server_key_iv: "d58ac3c935bfc7016dd10536c630bb03"
server_key_version: "encryptionkey@0"
I20260929 10:28:25.539534 25786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance:
uuid: "8d52fd55ae0d483cae0ea5723d7ed696"
format_stamp: "Formatted at 2026-09-29 10:28:25 on dist-test-slave-9cj5"
server_key: "6b144fac7a8089fe8b98d41af99c2c1e"
server_key_iv: "d58ac3c935bfc7016dd10536c630bb03"
server_key_version: "encryptionkey@0"
I20260929 10:28:25.543279 25786 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:28:25.546268 25800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:25.547591 25786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 10:28:25.547732 25786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "8d52fd55ae0d483cae0ea5723d7ed696"
format_stamp: "Formatted at 2026-09-29 10:28:25 on dist-test-slave-9cj5"
server_key: "6b144fac7a8089fe8b98d41af99c2c1e"
server_key_iv: "d58ac3c935bfc7016dd10536c630bb03"
server_key_version: "encryptionkey@0"
I20260929 10:28:25.547863 25786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:25.552666 25786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:25.553313 25786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:25.553500 25786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:25.554149 25786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:25.555083 25786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:25.555152 25786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:25.555223 25786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:25.555265 25786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:25.564256 25786 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:38603
I20260929 10:28:25.564321 25913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:38603 every 8 connection(s)
I20260929 10:28:25.565165 25786 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:28:25.568858 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 25786
I20260929 10:28:25.568988 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance
I20260929 10:28:25.569288 20848 external_mini_cluster.cc:1428] Setting key 413e658650aaa3d4a1b2fe30d3b60634
I20260929 10:28:25.571883 25503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:25.567209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:53731 (local address 127.20.92.62:34017)
0929 10:28:25.567351 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:25.567354 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:25.568364 (+ 1010us) server_negotiation.cc:407] Connection header received
0929 10:28:25.570018 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:25.570022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:25.570064 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:25.570156 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:25.571450 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:25.571454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:25.571456 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:28:25.571468 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:25.571498 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:28:25.571508 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:25.571596 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:25.571752 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 10:28:25.571787 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 10:28:25.572764 25916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:25.567566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:34017 (local address 127.20.92.3:53731)
0929 10:28:25.568196 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:25.568234 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:25.569707 (+ 1473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:25.570384 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:25.570392 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:25.570509 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:25.571175 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:28:25.571176 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:25.571215 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:28:25.571280 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:25.571599 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:25.571607 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:25.571613 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 10:28:25.571766 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 10:28:25.572006 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":114,"threads_started":1}
I20260929 10:28:25.573861 25914 heartbeater.cc:347] Connected to a master server at 127.20.92.62:34017
I20260929 10:28:25.574107 25914 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:25.574508 25914 heartbeater.cc:511] Master 127.20.92.62:34017 requested a full tablet report, sending...
I20260929 10:28:25.575489 25464 ts_manager.cc:194] Registered new tserver with Master: 8d52fd55ae0d483cae0ea5723d7ed696 (127.20.92.3:38603)
I20260929 10:28:25.583395 20848 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:28:25.587322 25503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:25.585425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42692 (local address 127.20.92.62:34017)
0929 10:28:25.585558 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:25.585562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:25.585714 (+ 152us) server_negotiation.cc:407] Connection header received
0929 10:28:25.585916 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:25.585919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:25.585955 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:25.586465 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:25.586965 (+ 500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:25.586968 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:25.586970 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:28:25.586979 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:25.586997 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:28:25.587005 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:25.587073 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:25.587180 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 10:28:25.587214 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 10:28:25.590458 25464 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:28:25.593362 25464 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:28:25.605497 25925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:25.602376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:38603 (local address 127.0.0.1:41196)
0929 10:28:25.603273 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:25.603325 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:25.603500 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:25.604022 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:25.604029 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:25.604100 (+ 71us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:25.604666 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:28:25.604669 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:25.604692 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:28:25.604773 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:25.605155 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:25.605164 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:25.605170 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 10:28:25.605287 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 10:28:25.605351 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":82,"threads_started":1}
I20260929 10:28:25.605508 25927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:25.602468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41196 (local address 127.20.92.3:38603)
0929 10:28:25.603023 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:25.603036 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:25.603415 (+ 379us) server_negotiation.cc:407] Connection header received
0929 10:28:25.603624 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:25.603632 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:25.603865 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:25.604005 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:25.604914 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:25.604922 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:25.604936 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:28:25.604952 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:25.604993 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:28:25.605051 (+ 58us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:25.605162 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:25.605318 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 10:28:25.605392 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":123,"threads_started":1}
I20260929 10:28:25.606287 25926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:25.602806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:33897 (local address 127.0.0.1:39402)
0929 10:28:25.603704 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:25.603717 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:25.603848 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:25.604386 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:25.604390 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:25.604405 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:25.604803 (+ 398us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:28:25.604804 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:25.604813 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:28:25.604860 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:25.605578 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:25.605581 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:25.605584 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 10:28:25.606132 (+ 548us) client_negotiation.cc:240] Negotiation successful
0929 10:28:25.606171 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"thread_start_us":90,"threads_started":1}
I20260929 10:28:25.606509 25929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:25.603023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48580 (local address 127.20.92.1:40189)
0929 10:28:25.603439 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:25.603452 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:25.603682 (+ 230us) server_negotiation.cc:407] Connection header received
0929 10:28:25.603866 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:25.603872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:25.604027 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:25.604140 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:25.605924 (+ 1784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:25.605929 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:25.605939 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:28:25.605954 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:25.605997 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:28:25.606029 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:25.606130 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:25.606294 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 10:28:25.606380 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":121,"threads_started":1}
I20260929 10:28:25.606693 25930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:25.603176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39402 (local address 127.20.92.2:33897)
0929 10:28:25.603723 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:25.603734 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:25.603877 (+ 143us) server_negotiation.cc:407] Connection header received
0929 10:28:25.603937 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:25.603944 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:25.604120 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:25.604260 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:25.605398 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:25.605403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:25.605413 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:28:25.605423 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:25.605457 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:28:25.605476 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:25.605790 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:25.606485 (+ 695us) server_negotiation.cc:299] Negotiation successful
0929 10:28:25.606550 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":146,"threads_started":1}
I20260929 10:28:25.606819 25928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:25.602409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:40189 (local address 127.0.0.1:48580)
0929 10:28:25.603593 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:25.603608 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:25.603736 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:25.604881 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:25.604884 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:25.604899 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:25.605676 (+ 777us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:28:25.605677 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:25.605688 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:28:25.605717 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:25.606169 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:25.606172 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:25.606175 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 10:28:25.606661 (+ 486us) client_negotiation.cc:240] Negotiation successful
0929 10:28:25.606695 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1113,"thread_start_us":90,"threads_started":1}
I20260929 10:28:25.609052 25848 tablet_service.cc:1467] Processing CreateTablet for tablet 5495b67a4d8244af811e85b5004c50f4 (DEFAULT_TABLE table=test-table [id=6f6ddc9c146d457ebac198586b4ec3b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:25.609781 25586 tablet_service.cc:1467] Processing CreateTablet for tablet 5495b67a4d8244af811e85b5004c50f4 (DEFAULT_TABLE table=test-table [id=6f6ddc9c146d457ebac198586b4ec3b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:25.609802 25717 tablet_service.cc:1467] Processing CreateTablet for tablet 5495b67a4d8244af811e85b5004c50f4 (DEFAULT_TABLE table=test-table [id=6f6ddc9c146d457ebac198586b4ec3b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:25.610562 25848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5495b67a4d8244af811e85b5004c50f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:25.610814 25586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5495b67a4d8244af811e85b5004c50f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:25.610832 25717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5495b67a4d8244af811e85b5004c50f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:25.617488 25931 tablet_bootstrap.cc:492] T 5495b67a4d8244af811e85b5004c50f4 P b5cf22164fcc4c658bd421338d86904f: Bootstrap starting.
I20260929 10:28:25.618768 25932 tablet_bootstrap.cc:492] T 5495b67a4d8244af811e85b5004c50f4 P 2e97dc86141d4c4790841c02662ec2f1: Bootstrap starting.
I20260929 10:28:25.619130 25933 tablet_bootstrap.cc:492] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696: Bootstrap starting.
I20260929 10:28:25.620436 25931 tablet_bootstrap.cc:654] T 5495b67a4d8244af811e85b5004c50f4 P b5cf22164fcc4c658bd421338d86904f: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:25.621063 25932 tablet_bootstrap.cc:654] T 5495b67a4d8244af811e85b5004c50f4 P 2e97dc86141d4c4790841c02662ec2f1: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:25.621171 25933 tablet_bootstrap.cc:654] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:25.621224 25931 log.cc:824] T 5495b67a4d8244af811e85b5004c50f4 P b5cf22164fcc4c658bd421338d86904f: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:25.622051 25932 log.cc:824] T 5495b67a4d8244af811e85b5004c50f4 P 2e97dc86141d4c4790841c02662ec2f1: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:25.622051 25933 log.cc:824] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:25.623325 25931 tablet_bootstrap.cc:492] T 5495b67a4d8244af811e85b5004c50f4 P b5cf22164fcc4c658bd421338d86904f: No bootstrap required, opened a new log
I20260929 10:28:25.623534 25931 ts_tablet_manager.cc:1397] T 5495b67a4d8244af811e85b5004c50f4 P b5cf22164fcc4c658bd421338d86904f: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260929 10:28:25.624058 25932 tablet_bootstrap.cc:492] T 5495b67a4d8244af811e85b5004c50f4 P 2e97dc86141d4c4790841c02662ec2f1: No bootstrap required, opened a new log
I20260929 10:28:25.624253 25932 ts_tablet_manager.cc:1397] T 5495b67a4d8244af811e85b5004c50f4 P 2e97dc86141d4c4790841c02662ec2f1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 10:28:25.624384 25933 tablet_bootstrap.cc:492] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696: No bootstrap required, opened a new log
I20260929 10:28:25.624570 25933 ts_tablet_manager.cc:1397] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 10:28:25.627144 25931 raft_consensus.cc:348] T 5495b67a4d8244af811e85b5004c50f4 P b5cf22164fcc4c658bd421338d86904f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d52fd55ae0d483cae0ea5723d7ed696" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38603 } } peers { permanent_uuid: "b5cf22164fcc4c658bd421338d86904f" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 40189 } } peers { permanent_uuid: "2e97dc86141d4c4790841c02662ec2f1" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 33897 } }
I20260929 10:28:25.627411 25931 raft_consensus.cc:374] T 5495b67a4d8244af811e85b5004c50f4 P b5cf22164fcc4c658bd421338d86904f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:25.627465 25931 raft_consensus.cc:729] T 5495b67a4d8244af811e85b5004c50f4 P b5cf22164fcc4c658bd421338d86904f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5cf22164fcc4c658bd421338d86904f, State: Initialized, Role: FOLLOWER
I20260929 10:28:25.627918 25931 consensus_queue.cc:260] T 5495b67a4d8244af811e85b5004c50f4 P b5cf22164fcc4c658bd421338d86904f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d52fd55ae0d483cae0ea5723d7ed696" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38603 } } peers { permanent_uuid: "b5cf22164fcc4c658bd421338d86904f" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 40189 } } peers { permanent_uuid: "2e97dc86141d4c4790841c02662ec2f1" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 33897 } }
I20260929 10:28:25.628564 25652 heartbeater.cc:503] Master 127.20.92.62:34017 was elected leader, sending a full tablet report...
I20260929 10:28:25.628931 25931 ts_tablet_manager.cc:1428] T 5495b67a4d8244af811e85b5004c50f4 P b5cf22164fcc4c658bd421338d86904f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 10:28:25.629309 25932 raft_consensus.cc:348] T 5495b67a4d8244af811e85b5004c50f4 P 2e97dc86141d4c4790841c02662ec2f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d52fd55ae0d483cae0ea5723d7ed696" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38603 } } peers { permanent_uuid: "b5cf22164fcc4c658bd421338d86904f" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 40189 } } peers { permanent_uuid: "2e97dc86141d4c4790841c02662ec2f1" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 33897 } }
I20260929 10:28:25.629309 25933 raft_consensus.cc:348] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d52fd55ae0d483cae0ea5723d7ed696" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38603 } } peers { permanent_uuid: "b5cf22164fcc4c658bd421338d86904f" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 40189 } } peers { permanent_uuid: "2e97dc86141d4c4790841c02662ec2f1" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 33897 } }
I20260929 10:28:25.629586 25933 raft_consensus.cc:374] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:25.629586 25932 raft_consensus.cc:374] T 5495b67a4d8244af811e85b5004c50f4 P 2e97dc86141d4c4790841c02662ec2f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:25.629653 25933 raft_consensus.cc:729] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d52fd55ae0d483cae0ea5723d7ed696, State: Initialized, Role: FOLLOWER
I20260929 10:28:25.629653 25932 raft_consensus.cc:729] T 5495b67a4d8244af811e85b5004c50f4 P 2e97dc86141d4c4790841c02662ec2f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e97dc86141d4c4790841c02662ec2f1, State: Initialized, Role: FOLLOWER
I20260929 10:28:25.630160 25932 consensus_queue.cc:260] T 5495b67a4d8244af811e85b5004c50f4 P 2e97dc86141d4c4790841c02662ec2f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d52fd55ae0d483cae0ea5723d7ed696" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38603 } } peers { permanent_uuid: "b5cf22164fcc4c658bd421338d86904f" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 40189 } } peers { permanent_uuid: "2e97dc86141d4c4790841c02662ec2f1" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 33897 } }
I20260929 10:28:25.630160 25933 consensus_queue.cc:260] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d52fd55ae0d483cae0ea5723d7ed696" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38603 } } peers { permanent_uuid: "b5cf22164fcc4c658bd421338d86904f" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 40189 } } peers { permanent_uuid: "2e97dc86141d4c4790841c02662ec2f1" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 33897 } }
I20260929 10:28:25.631193 25932 ts_tablet_manager.cc:1428] T 5495b67a4d8244af811e85b5004c50f4 P 2e97dc86141d4c4790841c02662ec2f1: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260929 10:28:25.631193 25933 ts_tablet_manager.cc:1428] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 10:28:25.631541 25783 heartbeater.cc:503] Master 127.20.92.62:34017 was elected leader, sending a full tablet report...
I20260929 10:28:25.632575 25914 heartbeater.cc:503] Master 127.20.92.62:34017 was elected leader, sending a full tablet report...
W20260929 10:28:25.635857 25784 tablet.cc:2369] T 5495b67a4d8244af811e85b5004c50f4 P 2e97dc86141d4c4790841c02662ec2f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:28:25.637373 25939 raft_consensus.cc:482] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:28:25.637537 25939 raft_consensus.cc:504] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d52fd55ae0d483cae0ea5723d7ed696" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38603 } } peers { permanent_uuid: "b5cf22164fcc4c658bd421338d86904f" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 40189 } } peers { permanent_uuid: "2e97dc86141d4c4790841c02662ec2f1" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 33897 } }
I20260929 10:28:25.638684 25939 leader_election.cc:288] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5cf22164fcc4c658bd421338d86904f (127.20.92.1:40189), 2e97dc86141d4c4790841c02662ec2f1 (127.20.92.2:33897)
I20260929 10:28:25.640789 25929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:25.638994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:40705 (local address 127.20.92.1:40189)
0929 10:28:25.639101 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:25.639105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:25.639142 (+ 37us) server_negotiation.cc:407] Connection header received
0929 10:28:25.639301 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:25.639307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:25.639348 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:25.639442 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:25.640294 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:25.640296 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:25.640299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:28:25.640309 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:25.640329 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:28:25.640337 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:25.640461 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:25.640596 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 10:28:25.640642 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 10:28:25.640834 25916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:25.638882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:40189 (local address 127.20.92.3:40705)
0929 10:28:25.639069 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:25.639089 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:25.639180 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:25.639551 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:25.639554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:25.639567 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:25.640091 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:28:25.640093 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:25.640103 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:28:25.640128 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:25.640494 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:25.640497 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:25.640499 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 10:28:25.640590 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 10:28:25.640631 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 10:28:25.641417 25930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:25.639304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:37167 (local address 127.20.92.2:33897)
0929 10:28:25.639417 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:25.639422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:25.639533 (+ 111us) server_negotiation.cc:407] Connection header received
0929 10:28:25.639645 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:25.639648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:25.639686 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:25.639810 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:25.640910 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:25.640913 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:25.640916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:28:25.640939 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:25.640963 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:28:25.640972 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:25.641032 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:25.641268 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 10:28:25.641308 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 10:28:25.641460 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:25.638900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:33897 (local address 127.20.92.3:37167)
0929 10:28:25.639457 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:25.639472 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:25.639556 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:25.640208 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:25.640210 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:25.640224 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:25.640738 (+ 514us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:28:25.640739 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:25.640749 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:28:25.640774 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:25.641177 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:25.641179 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:25.641181 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 10:28:25.641273 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 10:28:25.641307 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":24,"thread_start_us":116,"threads_started":1}
I20260929 10:28:25.641571 25606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5495b67a4d8244af811e85b5004c50f4" candidate_uuid: "8d52fd55ae0d483cae0ea5723d7ed696" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5cf22164fcc4c658bd421338d86904f" is_pre_election: true
I20260929 10:28:25.641916 25606 raft_consensus.cc:2454] T 5495b67a4d8244af811e85b5004c50f4 P b5cf22164fcc4c658bd421338d86904f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d52fd55ae0d483cae0ea5723d7ed696 in term 0.
I20260929 10:28:25.641999 25737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5495b67a4d8244af811e85b5004c50f4" candidate_uuid: "8d52fd55ae0d483cae0ea5723d7ed696" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e97dc86141d4c4790841c02662ec2f1" is_pre_election: true
I20260929 10:28:25.642253 25737 raft_consensus.cc:2454] T 5495b67a4d8244af811e85b5004c50f4 P 2e97dc86141d4c4790841c02662ec2f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d52fd55ae0d483cae0ea5723d7ed696 in term 0.
I20260929 10:28:25.642445 25801 leader_election.cc:302] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d52fd55ae0d483cae0ea5723d7ed696, b5cf22164fcc4c658bd421338d86904f; no voters:
I20260929 10:28:25.642710 25939 raft_consensus.cc:2781] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:28:25.642797 25939 raft_consensus.cc:482] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:28:25.642830 25939 raft_consensus.cc:3037] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:25.643865 25939 raft_consensus.cc:504] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d52fd55ae0d483cae0ea5723d7ed696" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38603 } } peers { permanent_uuid: "b5cf22164fcc4c658bd421338d86904f" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 40189 } } peers { permanent_uuid: "2e97dc86141d4c4790841c02662ec2f1" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 33897 } }
I20260929 10:28:25.644212 25939 leader_election.cc:288] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696 [CANDIDATE]: Term 1 election: Requested vote from peers b5cf22164fcc4c658bd421338d86904f (127.20.92.1:40189), 2e97dc86141d4c4790841c02662ec2f1 (127.20.92.2:33897)
I20260929 10:28:25.644619 25606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5495b67a4d8244af811e85b5004c50f4" candidate_uuid: "8d52fd55ae0d483cae0ea5723d7ed696" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5cf22164fcc4c658bd421338d86904f"
I20260929 10:28:25.644629 25737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5495b67a4d8244af811e85b5004c50f4" candidate_uuid: "8d52fd55ae0d483cae0ea5723d7ed696" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e97dc86141d4c4790841c02662ec2f1"
I20260929 10:28:25.644793 25606 raft_consensus.cc:3037] T 5495b67a4d8244af811e85b5004c50f4 P b5cf22164fcc4c658bd421338d86904f [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:25.644788 25737 raft_consensus.cc:3037] T 5495b67a4d8244af811e85b5004c50f4 P 2e97dc86141d4c4790841c02662ec2f1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:25.646117 25737 raft_consensus.cc:2454] T 5495b67a4d8244af811e85b5004c50f4 P 2e97dc86141d4c4790841c02662ec2f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d52fd55ae0d483cae0ea5723d7ed696 in term 1.
I20260929 10:28:25.646243 25606 raft_consensus.cc:2454] T 5495b67a4d8244af811e85b5004c50f4 P b5cf22164fcc4c658bd421338d86904f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d52fd55ae0d483cae0ea5723d7ed696 in term 1.
I20260929 10:28:25.646499 25803 leader_election.cc:302] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e97dc86141d4c4790841c02662ec2f1, 8d52fd55ae0d483cae0ea5723d7ed696; no voters:
I20260929 10:28:25.646709 25939 raft_consensus.cc:2781] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:25.646965 25939 raft_consensus.cc:686] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696 [term 1 LEADER]: Becoming Leader. State: Replica: 8d52fd55ae0d483cae0ea5723d7ed696, State: Running, Role: LEADER
I20260929 10:28:25.647301 25939 consensus_queue.cc:237] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d52fd55ae0d483cae0ea5723d7ed696" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38603 } } peers { permanent_uuid: "b5cf22164fcc4c658bd421338d86904f" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 40189 } } peers { permanent_uuid: "2e97dc86141d4c4790841c02662ec2f1" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 33897 } }
I20260929 10:28:25.650827 25464 catalog_manager.cc:5495] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d52fd55ae0d483cae0ea5723d7ed696 (127.20.92.3). New cstate: current_term: 1 leader_uuid: "8d52fd55ae0d483cae0ea5723d7ed696" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d52fd55ae0d483cae0ea5723d7ed696" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5cf22164fcc4c658bd421338d86904f" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 40189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e97dc86141d4c4790841c02662ec2f1" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 33897 } health_report { overall_health: UNKNOWN } } }
I20260929 10:28:25.670006 25927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:25.668368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41204 (local address 127.20.92.3:38603)
0929 10:28:25.668499 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:25.668503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:25.668516 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:28:25.668593 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:25.668595 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:25.668650 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:25.668723 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:25.669524 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:25.669527 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:25.669529 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:28:25.669541 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:25.669560 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:28:25.669568 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:25.669651 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:25.669842 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 10:28:25.669892 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 10:28:25.675877 25606 raft_consensus.cc:1261] T 5495b67a4d8244af811e85b5004c50f4 P b5cf22164fcc4c658bd421338d86904f [term 1 FOLLOWER]: Refusing update from remote peer 8d52fd55ae0d483cae0ea5723d7ed696: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:25.675881 25737 raft_consensus.cc:1261] T 5495b67a4d8244af811e85b5004c50f4 P 2e97dc86141d4c4790841c02662ec2f1 [term 1 FOLLOWER]: Refusing update from remote peer 8d52fd55ae0d483cae0ea5723d7ed696: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:25.676614 25941 consensus_queue.cc:1035] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e97dc86141d4c4790841c02662ec2f1" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 33897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:28:25.676811 25939 consensus_queue.cc:1035] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5cf22164fcc4c658bd421338d86904f" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 40189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 10:28:25.679515 25653 tablet.cc:2369] T 5495b67a4d8244af811e85b5004c50f4 P b5cf22164fcc4c658bd421338d86904f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:28:25.684814 25946 mvcc.cc:205] Tried to move back new op lower bound from 7334615882441691136 to 7334615882334294016. Current Snapshot: MvccSnapshot[applied={T|T < 7334615882441691136}]
I20260929 10:28:25.685509 25945 mvcc.cc:205] Tried to move back new op lower bound from 7334615882441691136 to 7334615882334294016. Current Snapshot: MvccSnapshot[applied={T|T < 7334615882441691136}]
I20260929 10:28:25.693673 25462 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:28:25.693883 25462 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:28:25.696292 25462 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ac70a4553bb45e7a7045e3d72d8396f: Sending DeleteTablet for 3 replicas of tablet 5495b67a4d8244af811e85b5004c50f4
I20260929 10:28:25.697041 25586 tablet_service.cc:1514] Processing DeleteTablet for tablet 5495b67a4d8244af811e85b5004c50f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:25 UTC) from {username='slave'} at 127.0.0.1:48580
I20260929 10:28:25.697094 25848 tablet_service.cc:1514] Processing DeleteTablet for tablet 5495b67a4d8244af811e85b5004c50f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:25 UTC) from {username='slave'} at 127.0.0.1:41196
I20260929 10:28:25.697175 25717 tablet_service.cc:1514] Processing DeleteTablet for tablet 5495b67a4d8244af811e85b5004c50f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:25 UTC) from {username='slave'} at 127.0.0.1:39402
I20260929 10:28:25.697507 25954 tablet_replica.cc:330] T 5495b67a4d8244af811e85b5004c50f4 P b5cf22164fcc4c658bd421338d86904f: stopping tablet replica
I20260929 10:28:25.697921 25954 raft_consensus.cc:2229] T 5495b67a4d8244af811e85b5004c50f4 P b5cf22164fcc4c658bd421338d86904f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:28:25.697948 25956 tablet_replica.cc:330] T 5495b67a4d8244af811e85b5004c50f4 P 2e97dc86141d4c4790841c02662ec2f1: stopping tablet replica
I20260929 10:28:25.698200 25956 raft_consensus.cc:2229] T 5495b67a4d8244af811e85b5004c50f4 P 2e97dc86141d4c4790841c02662ec2f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:28:25.698256 25954 raft_consensus.cc:2258] T 5495b67a4d8244af811e85b5004c50f4 P b5cf22164fcc4c658bd421338d86904f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:25.698451 25955 tablet_replica.cc:330] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696: stopping tablet replica
I20260929 10:28:25.698482 25956 raft_consensus.cc:2258] T 5495b67a4d8244af811e85b5004c50f4 P 2e97dc86141d4c4790841c02662ec2f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:25.698593 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 25524
I20260929 10:28:25.698889 25955 raft_consensus.cc:2229] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:28:25.699398 25955 raft_consensus.cc:2258] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:25.703080 25956 ts_tablet_manager.cc:1905] T 5495b67a4d8244af811e85b5004c50f4 P 2e97dc86141d4c4790841c02662ec2f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:25.705528 25956 ts_tablet_manager.cc:1918] T 5495b67a4d8244af811e85b5004c50f4 P 2e97dc86141d4c4790841c02662ec2f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:28:25.705662 25956 log.cc:1196] T 5495b67a4d8244af811e85b5004c50f4 P 2e97dc86141d4c4790841c02662ec2f1: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-1/wal/wals/5495b67a4d8244af811e85b5004c50f4
I20260929 10:28:25.706053 25956 ts_tablet_manager.cc:1939] T 5495b67a4d8244af811e85b5004c50f4 P 2e97dc86141d4c4790841c02662ec2f1: Deleting consensus metadata
W20260929 10:28:25.706498 25448 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.92.1:40189 (error 108)
I20260929 10:28:25.707472 25450 catalog_manager.cc:4841] TS 2e97dc86141d4c4790841c02662ec2f1 (127.20.92.2:33897): tablet 5495b67a4d8244af811e85b5004c50f4 (table test-table [id=6f6ddc9c146d457ebac198586b4ec3b2]) successfully deleted
I20260929 10:28:25.707948 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 25655
I20260929 10:28:25.707988 25955 ts_tablet_manager.cc:1905] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:25.708012 25928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:25.707716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:40189 (local address 127.0.0.1:48584)
0929 10:28:25.707845 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:25.707923 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.1:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 10:28:25.711944 25955 ts_tablet_manager.cc:1918] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260929 10:28:25.711946 25448 catalog_manager.cc:4584] TS b5cf22164fcc4c658bd421338d86904f (127.20.92.1:40189): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5495b67a4d8244af811e85b5004c50f4: Network error: Client connection negotiation failed: client connection to 127.20.92.1:40189: connect: Connection refused (error 111)
I20260929 10:28:25.712085 25955 log.cc:1196] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677679287822-20848-0/minicluster-data/ts-2/wal/wals/5495b67a4d8244af811e85b5004c50f4
I20260929 10:28:25.712412 25955 ts_tablet_manager.cc:1939] T 5495b67a4d8244af811e85b5004c50f4 P 8d52fd55ae0d483cae0ea5723d7ed696: Deleting consensus metadata
I20260929 10:28:25.713738 25451 catalog_manager.cc:4841] TS 8d52fd55ae0d483cae0ea5723d7ed696 (127.20.92.3:38603): tablet 5495b67a4d8244af811e85b5004c50f4 (table test-table [id=6f6ddc9c146d457ebac198586b4ec3b2]) successfully deleted
I20260929 10:28:25.717697 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 25786
I20260929 10:28:25.723456 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 25433
2026-09-29T10:28:25Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1008 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 10:28:25.744941 20848 mini_oidc.cc:152] Starting JWKS server
I20260929 10:28:25.746676 20848 webserver.cc:460] Webserver started at https://127.0.0.1:34919/ using document root <none> and password file <none>
I20260929 10:28:25.746871 20848 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 10:28:25.747151 20848 webserver.cc:460] Webserver started at http://127.0.0.1:46697/ using document root <none> and password file <none>
2026-09-29T10:28:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:28:25Z Disabled control of system clock
I20260929 10:28:25.756517 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:38019
--webserver_interface=127.20.92.62
--webserver_port=0
--builtin_ntp_servers=127.20.92.20:33493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:38019
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34919/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677679287822-20848-0/testchainca.cert with env {}
W20260929 10:28:25.872869 25963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:25.873185 25963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:25.873229 25963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:25.877367 25963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:28:25.877449 25963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:25.877465 25963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:25.877482 25963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:28:25.877498 25963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 10:28:25.877514 25963 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 10:28:25.877527 25963 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34919/jwks.json
I20260929 10:28:25.883153 25963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:33493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:38019
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:38019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34919/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677679287822-20848-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:25.884219 25963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:25.885210 25963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:25.894248 25969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:25.894248 25971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:25.894248 25968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:25.894418 25963 server_base.cc:1034] running on GCE node
I20260929 10:28:25.895282 25963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:25.896492 25963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:25.897727 25963 hybrid_clock.cc:648] HybridClock initialized: now 1790677705897653 us; error 43 us; skew 500 ppm
I20260929 10:28:25.899928 25963 webserver.cc:460] Webserver started at http://127.20.92.62:33677/ using document root <none> and password file <none>
I20260929 10:28:25.900556 25963 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:25.900628 25963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:25.900852 25963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:25.902988 25963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "06eb1f3505fb49df80aed181561ee9f4"
format_stamp: "Formatted at 2026-09-29 10:28:25 on dist-test-slave-9cj5"
server_key: "ca288b1f5a21b8d9617bc896e88b6eb7"
server_key_iv: "43b60535d3300500e8e5d93df4dd4218"
server_key_version: "encryptionkey@0"
I20260929 10:28:25.903573 25963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "06eb1f3505fb49df80aed181561ee9f4"
format_stamp: "Formatted at 2026-09-29 10:28:25 on dist-test-slave-9cj5"
server_key: "ca288b1f5a21b8d9617bc896e88b6eb7"
server_key_iv: "43b60535d3300500e8e5d93df4dd4218"
server_key_version: "encryptionkey@0"
I20260929 10:28:25.907507 25963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 10:28:25.910197 25977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:25.911582 25963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:28:25.911725 25963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "06eb1f3505fb49df80aed181561ee9f4"
format_stamp: "Formatted at 2026-09-29 10:28:25 on dist-test-slave-9cj5"
server_key: "ca288b1f5a21b8d9617bc896e88b6eb7"
server_key_iv: "43b60535d3300500e8e5d93df4dd4218"
server_key_version: "encryptionkey@0"
I20260929 10:28:25.911885 25963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:25.943213 25963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:25.943989 25963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:25.944203 25963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:25.951329 25963 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:38019
I20260929 10:28:25.951357 26029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:38019 every 8 connection(s)
I20260929 10:28:25.952311 25963 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:28:25.952950 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 25963
I20260929 10:28:25.953101 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:28:25.953477 20848 external_mini_cluster.cc:1428] Setting key e002a135700b92f34b51e2bcc2a1449d
I20260929 10:28:25.956118 26030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:25.961712 26030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06eb1f3505fb49df80aed181561ee9f4: Bootstrap starting.
I20260929 10:28:25.963706 26033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:25.954942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57128 (local address 127.20.92.62:38019)
0929 10:28:25.955723 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:25.955740 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:25.955779 (+ 39us) server_negotiation.cc:407] Connection header received
0929 10:28:25.956581 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:25.956600 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:25.957065 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:25.957437 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:25.957955 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:25.958992 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:25.959997 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:25.960528 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:25.960734 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:25.961848 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:25.961859 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:25.961876 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:28:25.962302 (+ 426us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:25.962359 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:28:25.962402 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:25.962522 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:25.962688 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 10:28:25.962992 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":132,"threads_started":1}
I20260929 10:28:25.964160 26030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06eb1f3505fb49df80aed181561ee9f4: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:25.965010 26030 log.cc:824] T 00000000000000000000000000000000 P 06eb1f3505fb49df80aed181561ee9f4: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:25.966769 20848 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:28:25.967252 26030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06eb1f3505fb49df80aed181561ee9f4: No bootstrap required, opened a new log
I20260929 10:28:25.971105 26030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06eb1f3505fb49df80aed181561ee9f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06eb1f3505fb49df80aed181561ee9f4" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38019 } }
I20260929 10:28:25.971422 26030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06eb1f3505fb49df80aed181561ee9f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:25.971505 26030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06eb1f3505fb49df80aed181561ee9f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06eb1f3505fb49df80aed181561ee9f4, State: Initialized, Role: FOLLOWER
I20260929 10:28:25.971958 26030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06eb1f3505fb49df80aed181561ee9f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06eb1f3505fb49df80aed181561ee9f4" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38019 } }
I20260929 10:28:25.972110 26030 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06eb1f3505fb49df80aed181561ee9f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:28:25.972183 26030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06eb1f3505fb49df80aed181561ee9f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:28:25.972282 26030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06eb1f3505fb49df80aed181561ee9f4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:25.973203 26030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06eb1f3505fb49df80aed181561ee9f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06eb1f3505fb49df80aed181561ee9f4" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38019 } }
I20260929 10:28:25.973548 26030 leader_election.cc:302] T 00000000000000000000000000000000 P 06eb1f3505fb49df80aed181561ee9f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06eb1f3505fb49df80aed181561ee9f4; no voters:
I20260929 10:28:25.973953 26030 leader_election.cc:288] T 00000000000000000000000000000000 P 06eb1f3505fb49df80aed181561ee9f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:28:25.974083 26035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06eb1f3505fb49df80aed181561ee9f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:25.974352 26035 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06eb1f3505fb49df80aed181561ee9f4 [term 1 LEADER]: Becoming Leader. State: Replica: 06eb1f3505fb49df80aed181561ee9f4, State: Running, Role: LEADER
I20260929 10:28:25.974635 26035 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06eb1f3505fb49df80aed181561ee9f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06eb1f3505fb49df80aed181561ee9f4" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38019 } }
I20260929 10:28:25.975234 26030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06eb1f3505fb49df80aed181561ee9f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:28:25.976364 26036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06eb1f3505fb49df80aed181561ee9f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06eb1f3505fb49df80aed181561ee9f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06eb1f3505fb49df80aed181561ee9f4" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38019 } } }
I20260929 10:28:25.976514 26036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06eb1f3505fb49df80aed181561ee9f4 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:25.976810 26037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06eb1f3505fb49df80aed181561ee9f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06eb1f3505fb49df80aed181561ee9f4. Latest consensus state: current_term: 1 leader_uuid: "06eb1f3505fb49df80aed181561ee9f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06eb1f3505fb49df80aed181561ee9f4" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38019 } } }
I20260929 10:28:25.976893 26044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:28:25.976917 26037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06eb1f3505fb49df80aed181561ee9f4 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:25.980017 26044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:28:25.986110 26044 catalog_manager.cc:1347] Generated new cluster ID: 58ab9793e9c945b796f62281e4316c03
I20260929 10:28:25.986198 26044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:28:26.022672 26044 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:28:26.023967 26044 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:28:26.035965 26044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06eb1f3505fb49df80aed181561ee9f4: Generated new TSK 0
I20260929 10:28:26.036768 26044 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 10:28:26.070040 26060 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 10:28:26.070168 26060 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 10:28:26.070788 26033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:26.052973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57134 (local address 127.20.92.62:38019)
0929 10:28:26.053119 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:26.053124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:26.053410 (+ 286us) server_negotiation.cc:407] Connection header received
0929 10:28:26.053601 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:26.053604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:26.053663 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:26.053825 (+ 162us) server_negotiation.cc:226] Negotiated authn=JWT
0929 10:28:26.054078 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:26.054901 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:26.055831 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:26.056173 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:26.056273 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:26.056591 (+ 318us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 10:28:26.070328 (+ 13737us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34919/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 10:28:26.070633 (+ 305us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57134: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34919/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 10:28:26.070863 26033 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57134: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34919/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 10:28:26.072052 26059 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:26.052919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38019 (local address 127.0.0.1:57134)
0929 10:28:26.053319 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:26.053332 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:26.053440 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:26.053843 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:26.053845 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:26.053857 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0929 10:28:26.053970 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:26.053975 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:26.055081 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:26.055084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:26.055710 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:26.055717 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:26.056286 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:26.056289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:26.056356 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:26.056403 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 10:28:26.070589 (+ 14186us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34919/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 10:28:26.070895 (+ 306us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.92.62:38019: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34919/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":117,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.20.92.62:38019: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34919/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 10:28:26.073309 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 25963
2026-09-29T10:28:26Z chronyd exiting
I20260929 10:28:26.111402 20848 test_util.cc:175] -----------------------------------------------
I20260929 10:28:26.111490 20848 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677679287822-20848-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (368 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 10:28:26.112504 20848 mini_oidc.cc:152] Starting JWKS server
I20260929 10:28:26.113374 20848 webserver.cc:460] Webserver started at https://127.0.0.1:33439/ using document root <none> and password file <none>
I20260929 10:28:26.113480 20848 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 10:28:26.113803 20848 webserver.cc:460] Webserver started at http://127.0.0.1:44603/ using document root <none> and password file <none>
2026-09-29T10:28:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:28:26Z Disabled control of system clock
I20260929 10:28:26.123375 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:33735
--webserver_interface=127.20.92.62
--webserver_port=0
--builtin_ntp_servers=127.20.92.20:37091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:33735
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33439/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677679287822-20848-0/testchainca.cert with env {}
W20260929 10:28:26.244696 26066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:26.245060 26066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:26.245141 26066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:26.250536 26066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:28:26.250679 26066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:26.250712 26066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:26.250756 26066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:28:26.250792 26066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 10:28:26.250828 26066 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 10:28:26.250859 26066 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33439/jwks.json
I20260929 10:28:26.256814 26066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:37091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:33735
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:33735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33439/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677679287822-20848-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:26.258059 26066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:26.259132 26066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:26.268121 26071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:26.268147 26072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:26.268143 26074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:26.268198 26066 server_base.cc:1034] running on GCE node
I20260929 10:28:26.268935 26066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:26.270233 26066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:26.271446 26066 hybrid_clock.cc:648] HybridClock initialized: now 1790677706271412 us; error 49 us; skew 500 ppm
I20260929 10:28:26.273532 26066 webserver.cc:460] Webserver started at http://127.20.92.62:34567/ using document root <none> and password file <none>
I20260929 10:28:26.274215 26066 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:26.274315 26066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:26.274547 26066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:26.276605 26066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "af71f093b4dd4694bd1a7aaa38df9c53"
format_stamp: "Formatted at 2026-09-29 10:28:26 on dist-test-slave-9cj5"
server_key: "618164ef7e4aec67764ef7a64139bd72"
server_key_iv: "f26712f3c92b19ff74c2cd0624a9e942"
server_key_version: "encryptionkey@0"
I20260929 10:28:26.277158 26066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "af71f093b4dd4694bd1a7aaa38df9c53"
format_stamp: "Formatted at 2026-09-29 10:28:26 on dist-test-slave-9cj5"
server_key: "618164ef7e4aec67764ef7a64139bd72"
server_key_iv: "f26712f3c92b19ff74c2cd0624a9e942"
server_key_version: "encryptionkey@0"
I20260929 10:28:26.280929 26066 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 10:28:26.283772 26080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:26.285127 26066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:28:26.285301 26066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "af71f093b4dd4694bd1a7aaa38df9c53"
format_stamp: "Formatted at 2026-09-29 10:28:26 on dist-test-slave-9cj5"
server_key: "618164ef7e4aec67764ef7a64139bd72"
server_key_iv: "f26712f3c92b19ff74c2cd0624a9e942"
server_key_version: "encryptionkey@0"
I20260929 10:28:26.285431 26066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:26.300762 26066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:26.301564 26066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:26.301827 26066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:26.309943 26066 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:33735
I20260929 10:28:26.309994 26132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:33735 every 8 connection(s)
I20260929 10:28:26.310977 26066 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:28:26.314406 26133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:26.320070 26133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af71f093b4dd4694bd1a7aaa38df9c53: Bootstrap starting.
I20260929 10:28:26.320269 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 26066
I20260929 10:28:26.320418 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:28:26.320730 20848 external_mini_cluster.cc:1428] Setting key 4bab4ec55460c64d5c64dd8c6b139758
I20260929 10:28:26.322786 26133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af71f093b4dd4694bd1a7aaa38df9c53: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:26.323668 26133 log.cc:824] T 00000000000000000000000000000000 P af71f093b4dd4694bd1a7aaa38df9c53: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:26.326068 26133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af71f093b4dd4694bd1a7aaa38df9c53: No bootstrap required, opened a new log
I20260929 10:28:26.330605 26133 raft_consensus.cc:348] T 00000000000000000000000000000000 P af71f093b4dd4694bd1a7aaa38df9c53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af71f093b4dd4694bd1a7aaa38df9c53" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 33735 } }
I20260929 10:28:26.330900 26133 raft_consensus.cc:374] T 00000000000000000000000000000000 P af71f093b4dd4694bd1a7aaa38df9c53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:26.330950 26133 raft_consensus.cc:729] T 00000000000000000000000000000000 P af71f093b4dd4694bd1a7aaa38df9c53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af71f093b4dd4694bd1a7aaa38df9c53, State: Initialized, Role: FOLLOWER
I20260929 10:28:26.331420 26133 consensus_queue.cc:260] T 00000000000000000000000000000000 P af71f093b4dd4694bd1a7aaa38df9c53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af71f093b4dd4694bd1a7aaa38df9c53" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 33735 } }
I20260929 10:28:26.331575 26133 raft_consensus.cc:388] T 00000000000000000000000000000000 P af71f093b4dd4694bd1a7aaa38df9c53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:28:26.331646 26133 raft_consensus.cc:482] T 00000000000000000000000000000000 P af71f093b4dd4694bd1a7aaa38df9c53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:28:26.331738 26133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af71f093b4dd4694bd1a7aaa38df9c53 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:26.331832 26136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:26.322241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58154 (local address 127.20.92.62:33735)
0929 10:28:26.322969 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:26.322990 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:26.323040 (+ 50us) server_negotiation.cc:407] Connection header received
0929 10:28:26.323910 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:26.323935 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:26.324388 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:26.324786 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:26.325226 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:26.326835 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:26.327833 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:26.328552 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:26.328719 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:26.329616 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:26.329625 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:26.329647 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:28:26.330200 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:26.330282 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:28:26.330309 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:26.330401 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:26.330559 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 10:28:26.330840 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":117,"threads_started":1}
I20260929 10:28:26.332723 26133 raft_consensus.cc:504] T 00000000000000000000000000000000 P af71f093b4dd4694bd1a7aaa38df9c53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af71f093b4dd4694bd1a7aaa38df9c53" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 33735 } }
I20260929 10:28:26.333086 26133 leader_election.cc:302] T 00000000000000000000000000000000 P af71f093b4dd4694bd1a7aaa38df9c53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af71f093b4dd4694bd1a7aaa38df9c53; no voters:
I20260929 10:28:26.333405 26133 leader_election.cc:288] T 00000000000000000000000000000000 P af71f093b4dd4694bd1a7aaa38df9c53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:28:26.333521 26138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af71f093b4dd4694bd1a7aaa38df9c53 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:26.333791 26138 raft_consensus.cc:686] T 00000000000000000000000000000000 P af71f093b4dd4694bd1a7aaa38df9c53 [term 1 LEADER]: Becoming Leader. State: Replica: af71f093b4dd4694bd1a7aaa38df9c53, State: Running, Role: LEADER
I20260929 10:28:26.334112 26138 consensus_queue.cc:237] T 00000000000000000000000000000000 P af71f093b4dd4694bd1a7aaa38df9c53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af71f093b4dd4694bd1a7aaa38df9c53" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 33735 } }
I20260929 10:28:26.335325 26133 sys_catalog.cc:564] T 00000000000000000000000000000000 P af71f093b4dd4694bd1a7aaa38df9c53 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:28:26.335650 20848 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:28:26.335785 26140 sys_catalog.cc:455] T 00000000000000000000000000000000 P af71f093b4dd4694bd1a7aaa38df9c53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af71f093b4dd4694bd1a7aaa38df9c53. Latest consensus state: current_term: 1 leader_uuid: "af71f093b4dd4694bd1a7aaa38df9c53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af71f093b4dd4694bd1a7aaa38df9c53" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 33735 } } }
I20260929 10:28:26.335909 26140 sys_catalog.cc:458] T 00000000000000000000000000000000 P af71f093b4dd4694bd1a7aaa38df9c53 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:26.336349 26142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:28:26.337425 26139 sys_catalog.cc:455] T 00000000000000000000000000000000 P af71f093b4dd4694bd1a7aaa38df9c53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af71f093b4dd4694bd1a7aaa38df9c53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af71f093b4dd4694bd1a7aaa38df9c53" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 33735 } } }
I20260929 10:28:26.337539 26139 sys_catalog.cc:458] T 00000000000000000000000000000000 P af71f093b4dd4694bd1a7aaa38df9c53 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:26.340133 26142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:28:26.346540 26142 catalog_manager.cc:1347] Generated new cluster ID: 2cef429ba5d041d58e1673b30783e14a
I20260929 10:28:26.346621 26142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:28:26.360700 26142 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:28:26.361675 26142 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:28:26.374027 26142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af71f093b4dd4694bd1a7aaa38df9c53: Generated new TSK 0
I20260929 10:28:26.374810 26142 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 10:28:26.406903 26163 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 10:28:26.407035 26163 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 10:28:26.407680 26136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:26.389479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58164 (local address 127.20.92.62:33735)
0929 10:28:26.389623 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:26.389628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:26.389948 (+ 320us) server_negotiation.cc:407] Connection header received
0929 10:28:26.390109 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:26.390112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:26.390195 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:26.390404 (+ 209us) server_negotiation.cc:226] Negotiated authn=JWT
0929 10:28:26.390714 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:26.391875 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:26.392866 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:26.393229 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:26.393337 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:26.393789 (+ 452us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 10:28:26.407201 (+ 13412us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33439/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 10:28:26.407528 (+ 327us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58164: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33439/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 10:28:26.407758 26136 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58164: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33439/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 10:28:26.407897 26162 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:26.389426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:33735 (local address 127.0.0.1:58164)
0929 10:28:26.389853 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:26.389869 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:26.389990 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:26.390412 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:26.390415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:26.390424 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 10:28:26.390590 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:26.390600 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:26.392011 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:26.392015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:26.392741 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:26.392749 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:26.393439 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:26.393443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:26.393529 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:26.393556 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 10:28:26.407468 (+ 13912us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33439/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 10:28:26.407633 (+ 165us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.92.62:33735: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33439/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":122,"threads_started":1}
I20260929 10:28:26.408715 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 26066
2026-09-29T10:28:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (336 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 10:28:26.449142 20848 mini_oidc.cc:152] Starting JWKS server
I20260929 10:28:26.450480 20848 webserver.cc:460] Webserver started at https://127.0.0.1:39397/ using document root <none> and password file <none>
I20260929 10:28:26.450582 20848 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 10:28:26.450830 20848 webserver.cc:460] Webserver started at http://127.0.0.1:33395/ using document root <none> and password file <none>
2026-09-29T10:28:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:28:26Z Disabled control of system clock
I20260929 10:28:26.460662 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:38611
--webserver_interface=127.20.92.62
--webserver_port=0
--builtin_ntp_servers=127.20.92.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:38611
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39397/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 10:28:26.581959 26169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:26.582224 26169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:26.582265 26169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:26.586760 26169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:28:26.586845 26169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:26.586871 26169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:26.586898 26169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:28:26.586932 26169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 10:28:26.586967 26169 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 10:28:26.586997 26169 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39397/jwks.json
I20260929 10:28:26.592263 26169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:38611
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:38611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39397/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:26.593397 26169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:26.594838 26169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:26.603718 26175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:26.603905 26169 server_base.cc:1034] running on GCE node
W20260929 10:28:26.603718 26174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:26.604105 26177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:26.604665 26169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:26.605914 26169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:26.607120 26169 hybrid_clock.cc:648] HybridClock initialized: now 1790677706607073 us; error 66 us; skew 500 ppm
I20260929 10:28:26.609164 26169 webserver.cc:460] Webserver started at http://127.20.92.62:44049/ using document root <none> and password file <none>
I20260929 10:28:26.609854 26169 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:26.609962 26169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:26.610208 26169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:26.612277 26169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "ad076150ce574ff3bb1fc1aeba336e82"
format_stamp: "Formatted at 2026-09-29 10:28:26 on dist-test-slave-9cj5"
server_key: "ef01df8ca8e8bbfeb6608bb80a623a23"
server_key_iv: "b45c35512ab56d74d244abd2957c51a6"
server_key_version: "encryptionkey@0"
I20260929 10:28:26.612852 26169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "ad076150ce574ff3bb1fc1aeba336e82"
format_stamp: "Formatted at 2026-09-29 10:28:26 on dist-test-slave-9cj5"
server_key: "ef01df8ca8e8bbfeb6608bb80a623a23"
server_key_iv: "b45c35512ab56d74d244abd2957c51a6"
server_key_version: "encryptionkey@0"
I20260929 10:28:26.616752 26169 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 10:28:26.619482 26183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:26.620728 26169 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260929 10:28:26.620886 26169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "ad076150ce574ff3bb1fc1aeba336e82"
format_stamp: "Formatted at 2026-09-29 10:28:26 on dist-test-slave-9cj5"
server_key: "ef01df8ca8e8bbfeb6608bb80a623a23"
server_key_iv: "b45c35512ab56d74d244abd2957c51a6"
server_key_version: "encryptionkey@0"
I20260929 10:28:26.621016 26169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:26.639132 26169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:26.640039 26169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:26.640268 26169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:26.647519 26169 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:38611
I20260929 10:28:26.647522 26235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:38611 every 8 connection(s)
I20260929 10:28:26.648555 26169 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:28:26.651825 26236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:26.657334 26236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad076150ce574ff3bb1fc1aeba336e82: Bootstrap starting.
I20260929 10:28:26.658392 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 26169
I20260929 10:28:26.658627 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:28:26.658944 20848 external_mini_cluster.cc:1428] Setting key c52bf5a682c291d49c4aa19220481009
I20260929 10:28:26.660254 26236 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad076150ce574ff3bb1fc1aeba336e82: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:26.661005 26236 log.cc:824] T 00000000000000000000000000000000 P ad076150ce574ff3bb1fc1aeba336e82: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:26.663303 26236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad076150ce574ff3bb1fc1aeba336e82: No bootstrap required, opened a new log
I20260929 10:28:26.667644 26236 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad076150ce574ff3bb1fc1aeba336e82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad076150ce574ff3bb1fc1aeba336e82" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38611 } }
I20260929 10:28:26.667896 26236 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad076150ce574ff3bb1fc1aeba336e82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:26.667958 26236 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad076150ce574ff3bb1fc1aeba336e82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad076150ce574ff3bb1fc1aeba336e82, State: Initialized, Role: FOLLOWER
I20260929 10:28:26.668435 26236 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad076150ce574ff3bb1fc1aeba336e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad076150ce574ff3bb1fc1aeba336e82" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38611 } }
I20260929 10:28:26.668598 26236 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad076150ce574ff3bb1fc1aeba336e82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:28:26.668678 26236 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad076150ce574ff3bb1fc1aeba336e82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:28:26.668809 26236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad076150ce574ff3bb1fc1aeba336e82 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:26.669477 26239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:26.660172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34362 (local address 127.20.92.62:38611)
0929 10:28:26.661016 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:26.661031 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:26.661068 (+ 37us) server_negotiation.cc:407] Connection header received
0929 10:28:26.662050 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:26.662072 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:26.662419 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:26.662705 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:26.663127 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:26.664214 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:26.665448 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:26.666055 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:26.666186 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:26.667216 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:26.667222 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:26.667235 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:28:26.667667 (+ 432us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:26.667775 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:28:26.667798 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:26.667964 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:26.668109 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 10:28:26.668466 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":182,"threads_started":1}
I20260929 10:28:26.669859 26236 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad076150ce574ff3bb1fc1aeba336e82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad076150ce574ff3bb1fc1aeba336e82" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38611 } }
I20260929 10:28:26.670207 26236 leader_election.cc:302] T 00000000000000000000000000000000 P ad076150ce574ff3bb1fc1aeba336e82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad076150ce574ff3bb1fc1aeba336e82; no voters:
I20260929 10:28:26.670567 26236 leader_election.cc:288] T 00000000000000000000000000000000 P ad076150ce574ff3bb1fc1aeba336e82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:28:26.670730 26241 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad076150ce574ff3bb1fc1aeba336e82 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:26.670960 26241 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad076150ce574ff3bb1fc1aeba336e82 [term 1 LEADER]: Becoming Leader. State: Replica: ad076150ce574ff3bb1fc1aeba336e82, State: Running, Role: LEADER
I20260929 10:28:26.671259 26241 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad076150ce574ff3bb1fc1aeba336e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad076150ce574ff3bb1fc1aeba336e82" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38611 } }
I20260929 10:28:26.671901 26236 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad076150ce574ff3bb1fc1aeba336e82 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:28:26.672574 20848 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:28:26.672574 26243 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad076150ce574ff3bb1fc1aeba336e82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad076150ce574ff3bb1fc1aeba336e82. Latest consensus state: current_term: 1 leader_uuid: "ad076150ce574ff3bb1fc1aeba336e82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad076150ce574ff3bb1fc1aeba336e82" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38611 } } }
I20260929 10:28:26.672778 26243 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad076150ce574ff3bb1fc1aeba336e82 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:26.673053 26242 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad076150ce574ff3bb1fc1aeba336e82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad076150ce574ff3bb1fc1aeba336e82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad076150ce574ff3bb1fc1aeba336e82" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38611 } } }
I20260929 10:28:26.673159 26242 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad076150ce574ff3bb1fc1aeba336e82 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:26.673662 26250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:28:26.676911 26250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:28:26.683004 26250 catalog_manager.cc:1347] Generated new cluster ID: 05102e5fbc064de6b0747ede6da9dc04
I20260929 10:28:26.683079 26250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:28:26.706355 26250 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:28:26.707450 26250 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:28:26.719549 26250 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad076150ce574ff3bb1fc1aeba336e82: Generated new TSK 0
I20260929 10:28:26.720409 26250 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:28:26.741139 26239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:26.735880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34370 (local address 127.20.92.62:38611)
0929 10:28:26.736005 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:26.736010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:26.736141 (+ 131us) server_negotiation.cc:407] Connection header received
0929 10:28:26.736281 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:26.736284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:26.736332 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:26.736431 (+ 99us) server_negotiation.cc:226] Negotiated authn=JWT
0929 10:28:26.736759 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:26.737309 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:26.738557 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:26.738844 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:26.738944 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:26.739275 (+ 331us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 10:28:26.740774 (+ 1499us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39397/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 10:28:26.741002 (+ 228us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34370: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39397/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":38}
W20260929 10:28:26.741214 26239 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34370: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39397/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 10:28:26.743116 26265 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:26.735703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38611 (local address 127.0.0.1:34370)
0929 10:28:26.736028 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:26.736043 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:26.736157 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:26.736465 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:26.736468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:26.736478 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0929 10:28:26.736608 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:26.736615 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:26.737415 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:26.737419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:26.738430 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:26.738440 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:26.738968 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:26.738971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:26.739060 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:26.739087 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 10:28:26.740944 (+ 1857us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39397/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 10:28:26.742924 (+ 1980us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.92.62:38611: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39397/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":1767,"thread_start_us":110,"threads_started":1}
I20260929 10:28:26.743709 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 26169
E20260929 10:28:26.751781 26266 webserver.cc:533] Webserver: sslize failed:
E20260929 10:28:26.751874 26266 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-29T10:28:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (333 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T10:28:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:28:26Z Disabled control of system clock
I20260929 10:28:26.791384 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:36133
--webserver_interface=127.20.92.62
--webserver_port=0
--builtin_ntp_servers=127.20.92.20:46133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:36133
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677679287822-20848-0/testchainca.cert with env {}
W20260929 10:28:26.908361 26270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:26.908666 26270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:26.908711 26270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:26.912886 26270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:28:26.912962 26270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:26.912978 26270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:26.912997 26270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:28:26.913012 26270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 10:28:26.913028 26270 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 10:28:26.913043 26270 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 10:28:26.918963 26270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:46133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:36133
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:36133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677679287822-20848-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:26.920048 26270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:26.921103 26270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:26.930071 26275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:26.930058 26276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:26.930116 26270 server_base.cc:1034] running on GCE node
W20260929 10:28:26.930063 26278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:26.930917 26270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:26.932144 26270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:26.933379 26270 hybrid_clock.cc:648] HybridClock initialized: now 1790677706933344 us; error 60 us; skew 500 ppm
I20260929 10:28:26.935587 26270 webserver.cc:460] Webserver started at http://127.20.92.62:46173/ using document root <none> and password file <none>
I20260929 10:28:26.936277 26270 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:26.936390 26270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:26.936641 26270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:26.939054 26270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "8bb1d6eaedb54ddd893b3201769b1d91"
format_stamp: "Formatted at 2026-09-29 10:28:26 on dist-test-slave-9cj5"
server_key: "59a63510a1b1e66569f6434ec6a7f297"
server_key_iv: "82d22e6a2134b6b0adeb3c79cde8a7e4"
server_key_version: "encryptionkey@0"
I20260929 10:28:26.939698 26270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "8bb1d6eaedb54ddd893b3201769b1d91"
format_stamp: "Formatted at 2026-09-29 10:28:26 on dist-test-slave-9cj5"
server_key: "59a63510a1b1e66569f6434ec6a7f297"
server_key_iv: "82d22e6a2134b6b0adeb3c79cde8a7e4"
server_key_version: "encryptionkey@0"
I20260929 10:28:26.943852 26270 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:28:26.946735 26284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:26.948246 26270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:28:26.948411 26270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "8bb1d6eaedb54ddd893b3201769b1d91"
format_stamp: "Formatted at 2026-09-29 10:28:26 on dist-test-slave-9cj5"
server_key: "59a63510a1b1e66569f6434ec6a7f297"
server_key_iv: "82d22e6a2134b6b0adeb3c79cde8a7e4"
server_key_version: "encryptionkey@0"
I20260929 10:28:26.948544 26270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:26.963280 26270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:26.964012 26270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:26.964224 26270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:26.971414 26270 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:36133
I20260929 10:28:26.971421 26336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:36133 every 8 connection(s)
I20260929 10:28:26.972357 26270 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:28:26.975314 26337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:26.977905 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 26270
I20260929 10:28:26.978021 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:28:26.978330 20848 external_mini_cluster.cc:1428] Setting key 738c1f3a8b9bcc4f43dc6964ec8dd8bd
I20260929 10:28:26.981611 26337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bb1d6eaedb54ddd893b3201769b1d91: Bootstrap starting.
I20260929 10:28:26.983878 26337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bb1d6eaedb54ddd893b3201769b1d91: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:26.984586 26337 log.cc:824] T 00000000000000000000000000000000 P 8bb1d6eaedb54ddd893b3201769b1d91: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:26.986460 26337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bb1d6eaedb54ddd893b3201769b1d91: No bootstrap required, opened a new log
I20260929 10:28:26.987506 26340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:26.979660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58310 (local address 127.20.92.62:36133)
0929 10:28:26.980326 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:26.980342 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:26.980377 (+ 35us) server_negotiation.cc:407] Connection header received
0929 10:28:26.981023 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:26.981042 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:26.981450 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:26.981802 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:26.982227 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:26.983234 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:26.984244 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:26.984727 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:26.984914 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:26.985788 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:26.985794 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:26.985819 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:28:26.986259 (+ 440us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:26.986313 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:28:26.986333 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:26.986441 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:26.986555 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 10:28:26.986819 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":132,"threads_started":1}
I20260929 10:28:26.990729 20848 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:28:26.990634 26337 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bb1d6eaedb54ddd893b3201769b1d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bb1d6eaedb54ddd893b3201769b1d91" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 36133 } }
I20260929 10:28:26.990945 26337 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bb1d6eaedb54ddd893b3201769b1d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:26.991021 26337 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bb1d6eaedb54ddd893b3201769b1d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bb1d6eaedb54ddd893b3201769b1d91, State: Initialized, Role: FOLLOWER
I20260929 10:28:26.991462 26337 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bb1d6eaedb54ddd893b3201769b1d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bb1d6eaedb54ddd893b3201769b1d91" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 36133 } }
I20260929 10:28:26.991626 26337 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bb1d6eaedb54ddd893b3201769b1d91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:28:26.991699 26337 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bb1d6eaedb54ddd893b3201769b1d91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:28:26.991820 26337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bb1d6eaedb54ddd893b3201769b1d91 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:26.992849 26337 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bb1d6eaedb54ddd893b3201769b1d91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bb1d6eaedb54ddd893b3201769b1d91" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 36133 } }
I20260929 10:28:26.993134 26337 leader_election.cc:302] T 00000000000000000000000000000000 P 8bb1d6eaedb54ddd893b3201769b1d91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bb1d6eaedb54ddd893b3201769b1d91; no voters:
I20260929 10:28:26.993379 26337 leader_election.cc:288] T 00000000000000000000000000000000 P 8bb1d6eaedb54ddd893b3201769b1d91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:28:26.993512 26342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bb1d6eaedb54ddd893b3201769b1d91 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:26.993777 26342 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bb1d6eaedb54ddd893b3201769b1d91 [term 1 LEADER]: Becoming Leader. State: Replica: 8bb1d6eaedb54ddd893b3201769b1d91, State: Running, Role: LEADER
I20260929 10:28:26.994143 26342 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bb1d6eaedb54ddd893b3201769b1d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bb1d6eaedb54ddd893b3201769b1d91" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 36133 } }
I20260929 10:28:26.994877 26337 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bb1d6eaedb54ddd893b3201769b1d91 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:28:26.995949 26343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bb1d6eaedb54ddd893b3201769b1d91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bb1d6eaedb54ddd893b3201769b1d91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bb1d6eaedb54ddd893b3201769b1d91" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 36133 } } }
I20260929 10:28:26.995925 26344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bb1d6eaedb54ddd893b3201769b1d91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bb1d6eaedb54ddd893b3201769b1d91. Latest consensus state: current_term: 1 leader_uuid: "8bb1d6eaedb54ddd893b3201769b1d91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bb1d6eaedb54ddd893b3201769b1d91" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 36133 } } }
I20260929 10:28:26.996083 26344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bb1d6eaedb54ddd893b3201769b1d91 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:26.996076 26343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bb1d6eaedb54ddd893b3201769b1d91 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:26.996986 26351 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:28:27.000293 26351 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:28:27.006405 26351 catalog_manager.cc:1347] Generated new cluster ID: 11e4ebfcd4324d0a9384bc875e5e3c6b
I20260929 10:28:27.006506 26351 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:28:27.035149 26351 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:28:27.036125 26351 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:28:27.042520 26351 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bb1d6eaedb54ddd893b3201769b1d91: Generated new TSK 0
I20260929 10:28:27.043280 26351 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:28:27.123687 26340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:27.054839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58324 (local address 127.20.92.62:36133)
0929 10:28:27.054996 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:27.055003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:27.055157 (+ 154us) server_negotiation.cc:407] Connection header received
0929 10:28:27.055337 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:27.055339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:27.055392 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:27.055487 (+ 95us) server_negotiation.cc:226] Negotiated authn=JWT
0929 10:28:27.055778 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:27.056344 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:27.057602 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:27.057935 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:27.058024 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:27.058412 (+ 388us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 10:28:27.123298 (+ 64886us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 10:28:27.123566 (+ 268us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58324: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 10:28:27.123790 26340 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58324: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 10:28:27.124061 26366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:27.054668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:36133 (local address 127.0.0.1:58324)
0929 10:28:27.055038 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:27.055060 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:27.055190 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:27.055522 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:27.055525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:27.055538 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0929 10:28:27.055662 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:27.055672 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:27.056468 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:27.056472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:27.057461 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:27.057471 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:27.058065 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:27.058069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:27.058171 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:27.058201 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 10:28:27.123571 (+ 65370us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 10:28:27.123828 (+ 257us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.92.62:36133: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":133,"threads_started":1}
I20260929 10:28:27.124950 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 26270
2026-09-29T10:28:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (362 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (133 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (136 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:28:27 dist-test-slave-9cj5 krb5kdc[26378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:28:27 dist-test-slave-9cj5 krb5kdc[26378](info): set up 2 sockets
Sep 29 10:28:27 dist-test-slave-9cj5 krb5kdc[26378](info): commencing operation
krb5kdc: starting...
W20260929 10:28:28.604171 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.188s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:28:28 dist-test-slave-9cj5 krb5kdc[26378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:28:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:28:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
I20260929 10:28:28.748766 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:35799
--webserver_interface=127.20.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:43457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:35799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:28.875293 26391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:28.875595 26391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:28.875638 26391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:28.880285 26391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:28:28.880380 26391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:28.880401 26391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:28.880419 26391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:28:28.880435 26391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:28:28.886325 26391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:43457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:35799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:35799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:28.887673 26391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:28.888983 26391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:28.899677 26397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:28.899677 26396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:28.899684 26399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:28.899746 26391 server_base.cc:1034] running on GCE node
I20260929 10:28:28.900815 26391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:28.902228 26391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:28.903586 26391 hybrid_clock.cc:648] HybridClock initialized: now 1790677708903564 us; error 35 us; skew 500 ppm
Sep 29 10:28:28 dist-test-slave-9cj5 krb5kdc[26378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677708, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:28.907526 26391 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:28:28.908744 26391 webserver.cc:460] Webserver started at http://127.20.92.62:46353/ using document root <none> and password file <none>
I20260929 10:28:28.909409 26391 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:28.909487 26391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:28.909833 26391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:28.912377 26391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "0e291cc23f6448998c50ea49d5991216"
format_stamp: "Formatted at 2026-09-29 10:28:28 on dist-test-slave-9cj5"
server_key: "0a612e833f00b975bead2961679254cc"
server_key_iv: "f6e6ce4580bab4cb1ddbfb7878cd141b"
server_key_version: "encryptionkey@0"
I20260929 10:28:28.913098 26391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "0e291cc23f6448998c50ea49d5991216"
format_stamp: "Formatted at 2026-09-29 10:28:28 on dist-test-slave-9cj5"
server_key: "0a612e833f00b975bead2961679254cc"
server_key_iv: "f6e6ce4580bab4cb1ddbfb7878cd141b"
server_key_version: "encryptionkey@0"
I20260929 10:28:28.918102 26391 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260929 10:28:28.921327 26406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:28.923563 26391 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 10:28:28.923767 26391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "0e291cc23f6448998c50ea49d5991216"
format_stamp: "Formatted at 2026-09-29 10:28:28 on dist-test-slave-9cj5"
server_key: "0a612e833f00b975bead2961679254cc"
server_key_iv: "f6e6ce4580bab4cb1ddbfb7878cd141b"
server_key_version: "encryptionkey@0"
I20260929 10:28:28.923938 26391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:28.940208 26391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:28.944307 26391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:28.944581 26391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:28.953636 26391 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:35799
I20260929 10:28:28.953678 26458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:35799 every 8 connection(s)
I20260929 10:28:28.954912 26391 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:28:28.956604 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 26391
I20260929 10:28:28.956765 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:28:28.957058 20848 external_mini_cluster.cc:1428] Setting key 204b04a9152a935f9487034b4db87ee6
I20260929 10:28:28.958309 26459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:28.965816 26459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e291cc23f6448998c50ea49d5991216: Bootstrap starting.
Sep 29 10:28:28 dist-test-slave-9cj5 krb5kdc[26378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:28.968467 26459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e291cc23f6448998c50ea49d5991216: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:28.969548 26459 log.cc:824] T 00000000000000000000000000000000 P 0e291cc23f6448998c50ea49d5991216: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:28.972271 26459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e291cc23f6448998c50ea49d5991216: No bootstrap required, opened a new log
I20260929 10:28:28.974818 26462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:28.958709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35000 (local address 127.20.92.62:35799)
0929 10:28:28.959166 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:28.959194 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:28.959237 (+ 43us) server_negotiation.cc:407] Connection header received
0929 10:28:28.960118 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:28.960141 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:28.960581 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:28.960972 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:28.962051 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:28.963530 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:28.964518 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:28.965261 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:28.965419 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:28.968503 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:28.968511 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:28.968531 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:28.968572 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:28.971083 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:28.971684 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:28.971690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:28.971698 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:28.971854 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:28.972206 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:28.972210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:28.972212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:28.972846 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:28.973066 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:28.973380 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 10:28:28.973713 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":136,"threads_started":1}
I20260929 10:28:28.977342 26459 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e291cc23f6448998c50ea49d5991216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e291cc23f6448998c50ea49d5991216" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 35799 } }
I20260929 10:28:28.977641 26459 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e291cc23f6448998c50ea49d5991216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:28.977684 26459 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e291cc23f6448998c50ea49d5991216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e291cc23f6448998c50ea49d5991216, State: Initialized, Role: FOLLOWER
I20260929 10:28:28.978219 26459 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e291cc23f6448998c50ea49d5991216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e291cc23f6448998c50ea49d5991216" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 35799 } }
I20260929 10:28:28.978359 26459 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e291cc23f6448998c50ea49d5991216 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:28:28.978391 26459 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e291cc23f6448998c50ea49d5991216 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:28:28.978516 26459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e291cc23f6448998c50ea49d5991216 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:28.979569 26459 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e291cc23f6448998c50ea49d5991216 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e291cc23f6448998c50ea49d5991216" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 35799 } }
I20260929 10:28:28.979974 26459 leader_election.cc:302] T 00000000000000000000000000000000 P 0e291cc23f6448998c50ea49d5991216 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e291cc23f6448998c50ea49d5991216; no voters:
I20260929 10:28:28.979991 26464 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e291cc23f6448998c50ea49d5991216 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:28:28.980243 26464 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e291cc23f6448998c50ea49d5991216 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e291cc23f6448998c50ea49d5991216" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 35799 } }
I20260929 10:28:28.980319 26459 leader_election.cc:288] T 00000000000000000000000000000000 P 0e291cc23f6448998c50ea49d5991216 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:28:28.980468 26465 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e291cc23f6448998c50ea49d5991216 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:28.980716 26465 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e291cc23f6448998c50ea49d5991216 [term 1 LEADER]: Becoming Leader. State: Replica: 0e291cc23f6448998c50ea49d5991216, State: Running, Role: LEADER
I20260929 10:28:28.981036 26465 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e291cc23f6448998c50ea49d5991216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e291cc23f6448998c50ea49d5991216" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 35799 } }
I20260929 10:28:28.981272 26464 leader_election.cc:302] T 00000000000000000000000000000000 P 0e291cc23f6448998c50ea49d5991216 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e291cc23f6448998c50ea49d5991216; no voters:
I20260929 10:28:28.981459 26464 leader_election.cc:288] T 00000000000000000000000000000000 P 0e291cc23f6448998c50ea49d5991216 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers
I20260929 10:28:28.981839 26459 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e291cc23f6448998c50ea49d5991216 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:28:28.982400 26464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e291cc23f6448998c50ea49d5991216 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e291cc23f6448998c50ea49d5991216. Latest consensus state: current_term: 1 leader_uuid: "0e291cc23f6448998c50ea49d5991216" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e291cc23f6448998c50ea49d5991216" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 35799 } } }
I20260929 10:28:28.982542 26464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e291cc23f6448998c50ea49d5991216 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:28.982694 26466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e291cc23f6448998c50ea49d5991216 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e291cc23f6448998c50ea49d5991216" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e291cc23f6448998c50ea49d5991216" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 35799 } } }
I20260929 10:28:28.982885 26466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e291cc23f6448998c50ea49d5991216 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:28.983112 26471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:28:28.986600 26471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:28:28.992915 26471 catalog_manager.cc:1347] Generated new cluster ID: 5c1212d949e84ff1b64fd938d80acf6c
I20260929 10:28:28.993007 26471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:28:29.013818 26471 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:28:29.015074 26471 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:28:29.026513 26471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e291cc23f6448998c50ea49d5991216: Generated new TSK 0
I20260929 10:28:29.027498 26471 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
I20260929 10:28:29.074672 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:0
--local_ip_for_outbound_sockets=127.20.92.1
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:35799
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:43457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:29.201685 26487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:29.202075 26487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:29.202169 26487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:29.206521 26487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:29.206620 26487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:29.206719 26487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:28:29.212499 26487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:43457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:35799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:29.213678 26487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:29.214918 26487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:29.226097 26492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:29.226116 26495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:29.226111 26493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:29.226406 26487 server_base.cc:1034] running on GCE node
I20260929 10:28:29.226900 26487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:29.227644 26487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:29.228900 26487 hybrid_clock.cc:648] HybridClock initialized: now 1790677709228879 us; error 28 us; skew 500 ppm
Sep 29 10:28:29 dist-test-slave-9cj5 krb5kdc[26378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677709, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:29.235526 26487 init.cc:377] Logged in from keytab as kudu/127.20.92.1@KRBTEST.COM (short username kudu)
I20260929 10:28:29.236618 26487 webserver.cc:460] Webserver started at http://127.20.92.1:44001/ using document root <none> and password file <none>
I20260929 10:28:29.237226 26487 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:29.237293 26487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:29.237520 26487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:29.239727 26487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-0/data/instance:
uuid: "6d1748d6f12e45b2b5452afb29f0fb2c"
format_stamp: "Formatted at 2026-09-29 10:28:29 on dist-test-slave-9cj5"
server_key: "b150e875f644a254c24c1d8697e60224"
server_key_iv: "1d2a3e7eec53c67a77f11f8512800397"
server_key_version: "encryptionkey@0"
I20260929 10:28:29.240355 26487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance:
uuid: "6d1748d6f12e45b2b5452afb29f0fb2c"
format_stamp: "Formatted at 2026-09-29 10:28:29 on dist-test-slave-9cj5"
server_key: "b150e875f644a254c24c1d8697e60224"
server_key_iv: "1d2a3e7eec53c67a77f11f8512800397"
server_key_version: "encryptionkey@0"
I20260929 10:28:29.244434 26487 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:28:29.247313 26502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:29.249004 26487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260929 10:28:29.249220 26487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "6d1748d6f12e45b2b5452afb29f0fb2c"
format_stamp: "Formatted at 2026-09-29 10:28:29 on dist-test-slave-9cj5"
server_key: "b150e875f644a254c24c1d8697e60224"
server_key_iv: "1d2a3e7eec53c67a77f11f8512800397"
server_key_version: "encryptionkey@0"
I20260929 10:28:29.249392 26487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:29.261624 26487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:29.265323 26487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:29.265545 26487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:29.266319 26487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:29.267180 26487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:29.267253 26487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:29.267338 26487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:29.267412 26487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:29.277462 26487 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:45301
I20260929 10:28:29.277541 26615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:45301 every 8 connection(s)
I20260929 10:28:29.278508 26487 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
I20260929 10:28:29.282616 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 26487
I20260929 10:28:29.282928 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance
I20260929 10:28:29.283294 20848 external_mini_cluster.cc:1428] Setting key 9b7ac25fdc6e887ee86637acbdcc280e
Sep 29 10:28:29 dist-test-slave-9cj5 krb5kdc[26378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677709, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.92.2@KRBTEST.COM; defaulting to no policy
I20260929 10:28:29.296281 26462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:29.280563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:42971 (local address 127.20.92.62:35799)
0929 10:28:29.280743 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:29.280746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:29.281605 (+ 859us) server_negotiation.cc:407] Connection header received
0929 10:28:29.283868 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:29.283873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:29.283942 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:29.284030 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:29.285788 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:29.286572 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:29.287785 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:29.288286 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:29.288424 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:29.291533 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:29.291537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:29.291541 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:29.291580 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:29.293529 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:29.294190 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:29.294196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:29.294201 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:29.294301 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:29.294866 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:29.294869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:29.294871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:29.295077 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:29.295212 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:29.295937 (+ 725us) server_negotiation.cc:299] Negotiation successful
0929 10:28:29.296087 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 10:28:29.297365 26618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:29.280924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:35799 (local address 127.20.92.1:42971)
0929 10:28:29.281424 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:29.281471 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:29.283434 (+ 1963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:29.284242 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:29.284250 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:29.284806 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:29.285550 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:29.285575 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:29.286722 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:29.286724 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:29.287663 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:29.287672 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:29.288453 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:29.288456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:29.288609 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:29.289307 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:29.289337 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:29.291338 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:29.293672 (+ 2334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:29.293678 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:29.293692 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:29.294049 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:29.294443 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:29.294446 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:29.294448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:29.294604 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:29.295243 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:29.295254 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:29.295655 (+ 401us) client_negotiation.cc:769] Sending connection context
0929 10:28:29.295873 (+ 218us) client_negotiation.cc:240] Negotiation successful
0929 10:28:29.296248 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.2@KRBTEST.COM" created.
I20260929 10:28:29.298936 26616 heartbeater.cc:347] Connected to a master server at 127.20.92.62:35799
I20260929 10:28:29.299214 26616 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:29.299731 26616 heartbeater.cc:511] Master 127.20.92.62:35799 requested a full tablet report, sending...
I20260929 10:28:29.301261 26423 ts_manager.cc:194] Registered new tserver with Master: 6d1748d6f12e45b2b5452afb29f0fb2c (127.20.92.1:45301)
I20260929 10:28:29.302443 26423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.1@KRBTEST.COM'} at 127.20.92.1:42971
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
I20260929 10:28:29.317670 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:0
--local_ip_for_outbound_sockets=127.20.92.2
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:35799
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:43457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:29.436403 26621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:29.436738 26621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:29.436789 26621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:29.441243 26621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:29.441325 26621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:29.441429 26621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:28:29.447209 26621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:43457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:35799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:29.448451 26621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:29.449433 26621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:29.460317 26626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:29.460351 26627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:29.460480 26621 server_base.cc:1034] running on GCE node
W20260929 10:28:29.460711 26629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:29.461205 26621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:29.462057 26621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:29.463240 26621 hybrid_clock.cc:648] HybridClock initialized: now 1790677709463214 us; error 44 us; skew 500 ppm
Sep 29 10:28:29 dist-test-slave-9cj5 krb5kdc[26378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677709, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:29.466308 26621 init.cc:377] Logged in from keytab as kudu/127.20.92.2@KRBTEST.COM (short username kudu)
I20260929 10:28:29.467248 26621 webserver.cc:460] Webserver started at http://127.20.92.2:40625/ using document root <none> and password file <none>
I20260929 10:28:29.467796 26621 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:29.467857 26621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:29.468022 26621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:29.470172 26621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-1/data/instance:
uuid: "ddb16e600039491ea7d85aaf7bd00234"
format_stamp: "Formatted at 2026-09-29 10:28:29 on dist-test-slave-9cj5"
server_key: "eb3a7377a0a9408b54a7670845d944ef"
server_key_iv: "6cc7105330a52a04dadb1f1ddf684de6"
server_key_version: "encryptionkey@0"
I20260929 10:28:29.470789 26621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance:
uuid: "ddb16e600039491ea7d85aaf7bd00234"
format_stamp: "Formatted at 2026-09-29 10:28:29 on dist-test-slave-9cj5"
server_key: "eb3a7377a0a9408b54a7670845d944ef"
server_key_iv: "6cc7105330a52a04dadb1f1ddf684de6"
server_key_version: "encryptionkey@0"
I20260929 10:28:29.474908 26621 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 10:28:29.477533 26636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:29.479000 26621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 10:28:29.479202 26621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "ddb16e600039491ea7d85aaf7bd00234"
format_stamp: "Formatted at 2026-09-29 10:28:29 on dist-test-slave-9cj5"
server_key: "eb3a7377a0a9408b54a7670845d944ef"
server_key_iv: "6cc7105330a52a04dadb1f1ddf684de6"
server_key_version: "encryptionkey@0"
I20260929 10:28:29.479346 26621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:29.496513 26621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:29.499924 26621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:29.500135 26621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:29.500746 26621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:29.501578 26621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:29.501647 26621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:29.501721 26621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:29.501784 26621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:29.512339 26621 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:38617
I20260929 10:28:29.512367 26749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:38617 every 8 connection(s)
I20260929 10:28:29.513271 26621 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
I20260929 10:28:29.514681 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 26621
I20260929 10:28:29.514775 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance
I20260929 10:28:29.515192 20848 external_mini_cluster.cc:1428] Setting key c110595d8a836aa17e8d4d226ff36ec5
Sep 29 10:28:29 dist-test-slave-9cj5 krb5kdc[26378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677709, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.92.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.3@KRBTEST.COM" created.
I20260929 10:28:29.530875 26462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:29.515249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:44283 (local address 127.20.92.62:35799)
0929 10:28:29.515421 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:29.515426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:29.516616 (+ 1190us) server_negotiation.cc:407] Connection header received
0929 10:28:29.518577 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:29.518580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:29.518626 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:29.518702 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:29.520007 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:29.521032 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:29.522451 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:29.522904 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:29.523043 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:29.526210 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:29.526214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:29.526216 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:29.526249 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:29.527836 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:29.528610 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:29.528613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:29.528615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:29.528720 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:29.529387 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:29.529397 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:29.529398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:29.529648 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:29.529803 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:29.530451 (+ 648us) server_negotiation.cc:299] Negotiation successful
0929 10:28:29.530611 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 10:28:29.531682 26752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:29.515680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:35799 (local address 127.20.92.2:44283)
0929 10:28:29.516431 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:29.516489 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:29.518298 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:29.518899 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:29.518912 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:29.519336 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:29.519846 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:29.519861 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:29.521205 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:29.521209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:29.522281 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:29.522292 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:29.523084 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:29.523086 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:29.523255 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:29.524078 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:29.524094 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:29.526005 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:29.528011 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:29.528017 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:29.528053 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:29.528454 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:29.528892 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:29.528898 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:29.528900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:29.529256 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:29.529844 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:29.529855 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:29.530135 (+ 280us) client_negotiation.cc:769] Sending connection context
0929 10:28:29.530347 (+ 212us) client_negotiation.cc:240] Negotiation successful
0929 10:28:29.530630 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":183,"threads_started":1}
I20260929 10:28:29.533187 26750 heartbeater.cc:347] Connected to a master server at 127.20.92.62:35799
I20260929 10:28:29.533471 26750 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:29.534158 26750 heartbeater.cc:511] Master 127.20.92.62:35799 requested a full tablet report, sending...
I20260929 10:28:29.535419 26423 ts_manager.cc:194] Registered new tserver with Master: ddb16e600039491ea7d85aaf7bd00234 (127.20.92.2:38617)
I20260929 10:28:29.536064 26423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.2@KRBTEST.COM'} at 127.20.92.2:44283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
I20260929 10:28:29.550531 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:0
--local_ip_for_outbound_sockets=127.20.92.3
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:35799
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:43457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:29.664760 26755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:29.665055 26755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:29.665103 26755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:29.669413 26755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:29.669476 26755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:29.669574 26755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:28:29.675179 26755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:43457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:35799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:29.676303 26755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:29.677338 26755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:29.687605 26760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:29.687623 26761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:29.687604 26763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:29.688066 26755 server_base.cc:1034] running on GCE node
I20260929 10:28:29.688505 26755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:29.689184 26755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:29.690403 26755 hybrid_clock.cc:648] HybridClock initialized: now 1790677709690381 us; error 33 us; skew 500 ppm
Sep 29 10:28:29 dist-test-slave-9cj5 krb5kdc[26378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677709, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:29.693470 26755 init.cc:377] Logged in from keytab as kudu/127.20.92.3@KRBTEST.COM (short username kudu)
I20260929 10:28:29.694487 26755 webserver.cc:460] Webserver started at http://127.20.92.3:43351/ using document root <none> and password file <none>
I20260929 10:28:29.695084 26755 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:29.695178 26755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:29.695420 26755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:29.697614 26755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-2/data/instance:
uuid: "11df1962949c4803a7a667feb7b5b7d1"
format_stamp: "Formatted at 2026-09-29 10:28:29 on dist-test-slave-9cj5"
server_key: "70449a2d3dc4628aa59416620020556c"
server_key_iv: "2079d68de7b9eeedbdac2b7b8271a17a"
server_key_version: "encryptionkey@0"
I20260929 10:28:29.698261 26755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance:
uuid: "11df1962949c4803a7a667feb7b5b7d1"
format_stamp: "Formatted at 2026-09-29 10:28:29 on dist-test-slave-9cj5"
server_key: "70449a2d3dc4628aa59416620020556c"
server_key_iv: "2079d68de7b9eeedbdac2b7b8271a17a"
server_key_version: "encryptionkey@0"
I20260929 10:28:29.702131 26755 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 10:28:29.704726 26770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:29.706276 26755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:28:29.706444 26755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "11df1962949c4803a7a667feb7b5b7d1"
format_stamp: "Formatted at 2026-09-29 10:28:29 on dist-test-slave-9cj5"
server_key: "70449a2d3dc4628aa59416620020556c"
server_key_iv: "2079d68de7b9eeedbdac2b7b8271a17a"
server_key_version: "encryptionkey@0"
I20260929 10:28:29.706578 26755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:29.719964 26755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:29.724020 26755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:29.724248 26755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:29.724875 26755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:29.725811 26755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:29.725875 26755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:29.725941 26755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:29.725987 26755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:29.735527 26755 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:38709
I20260929 10:28:29.735581 26883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:38709 every 8 connection(s)
I20260929 10:28:29.736460 26755 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:28:29.737885 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 26755
I20260929 10:28:29.738052 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance
I20260929 10:28:29.738401 20848 external_mini_cluster.cc:1428] Setting key 5a6eb00717ee48a08fbe3c482a0a7f46
Sep 29 10:28:29 dist-test-slave-9cj5 krb5kdc[26378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677709, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:29.752416 26462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:29.738602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:44437 (local address 127.20.92.62:35799)
0929 10:28:29.738797 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:29.738801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:29.739986 (+ 1185us) server_negotiation.cc:407] Connection header received
0929 10:28:29.741584 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:29.741587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:29.741631 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:29.741729 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:29.743077 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:29.743717 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:29.744949 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:29.745180 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:29.745276 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:29.748144 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:29.748147 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:29.748150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:29.748188 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:29.749643 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:29.750347 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:29.750350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:29.750352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:29.750440 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:29.750969 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:29.750972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:29.750974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:29.751223 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:29.751310 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:29.752099 (+ 789us) server_negotiation.cc:299] Negotiation successful
0929 10:28:29.752245 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 10:28:29.753435 26886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:29.739238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:35799 (local address 127.20.92.3:44437)
0929 10:28:29.739831 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:29.739888 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:29.741359 (+ 1471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:29.742003 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:29.742010 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:29.742421 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:29.742921 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:29.742935 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:29.743857 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:29.743860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:29.744755 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:29.744764 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:29.745362 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:29.745366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:29.745529 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:29.746335 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:29.746353 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:29.747950 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:29.749858 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:29.749863 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:29.749877 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:29.750160 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:29.750604 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:29.750607 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:29.750608 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:29.750826 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:29.751456 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:29.751463 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:29.751750 (+ 287us) client_negotiation.cc:769] Sending connection context
0929 10:28:29.751981 (+ 231us) client_negotiation.cc:240] Negotiation successful
0929 10:28:29.752360 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":192,"threads_started":1}
I20260929 10:28:29.754988 26884 heartbeater.cc:347] Connected to a master server at 127.20.92.62:35799
I20260929 10:28:29.755260 26884 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:29.755796 26884 heartbeater.cc:511] Master 127.20.92.62:35799 requested a full tablet report, sending...
I20260929 10:28:29.756947 26423 ts_manager.cc:194] Registered new tserver with Master: 11df1962949c4803a7a667feb7b5b7d1 (127.20.92.3:38709)
I20260929 10:28:29.757550 26423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.3@KRBTEST.COM'} at 127.20.92.3:44437
I20260929 10:28:29.765419 20848 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/test-admin.keytab.
Sep 29 10:28:29 dist-test-slave-9cj5 krb5kdc[26378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:29.784744 20848 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 10:28:29.787757 26891 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:29.786288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37992 (local address 127.20.92.1:45301)
0929 10:28:29.786550 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:29.786565 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:29.786589 (+ 24us) server_negotiation.cc:407] Connection header received
0929 10:28:29.786675 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:29.786679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:29.786900 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:29.787018 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:29.787582 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37992: BlockingRecv error: recv got EOF from 127.0.0.1:37992 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":81,"threads_started":1}
W20260929 10:28:29.787801 20848 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/krb5cc' of type FILE
Sep 29 10:28:29 dist-test-slave-9cj5 krb5kdc[26378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:29.789191 20848 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 10:28:29 dist-test-slave-9cj5 krb5kdc[26378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:29.790151 20848 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 10:28:29 dist-test-slave-9cj5 krb5kdc[26378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.1@KRBTEST.COM
I20260929 10:28:29.800508 26891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:29.791083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38002 (local address 127.20.92.1:45301)
0929 10:28:29.791208 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:29.791213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:29.791355 (+ 142us) server_negotiation.cc:407] Connection header received
0929 10:28:29.791508 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:29.791516 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:29.791556 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:29.791642 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:29.792217 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:29.793155 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:29.794060 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:29.794450 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:29.794569 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:29.796905 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:29.796914 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:29.796928 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:29.796953 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:29.798677 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:29.799138 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:29.799144 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:29.799145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:29.799221 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:29.799506 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:29.799510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:29.799512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:29.799763 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:29.799981 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:29.800170 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 10:28:29.800330 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 10:28:29.805522 26891 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:29.804273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38014 (local address 127.20.92.1:45301)
0929 10:28:29.804393 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:29.804397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:29.804594 (+ 197us) server_negotiation.cc:407] Connection header received
0929 10:28:29.804790 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:29.804793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:29.804833 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:29.804924 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:29.805407 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38014: BlockingRecv error: recv got EOF from 127.0.0.1:38014 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260929 10:28:29.805914 20848 init.cc:244] Error while opening credential cache '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/krb5cc)
Sep 29 10:28:29 dist-test-slave-9cj5 krb5kdc[26378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:29.807410 20848 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 10:28:29 dist-test-slave-9cj5 krb5kdc[26378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:29.808473 20848 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 10:28:29 dist-test-slave-9cj5 krb5kdc[26378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.1@KRBTEST.COM
I20260929 10:28:29.818890 26891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:29.809542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38020 (local address 127.20.92.1:45301)
0929 10:28:29.809695 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:29.809700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:29.809878 (+ 178us) server_negotiation.cc:407] Connection header received
0929 10:28:29.810035 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:29.810038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:29.810089 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:29.810222 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:29.810774 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:29.811391 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:29.812273 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:29.812516 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:29.812630 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:29.815242 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:29.815246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:29.815248 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:29.815275 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:29.816985 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:29.817450 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:29.817455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:29.817457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:29.817549 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:29.817919 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:29.817924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:29.817926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:29.818124 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:29.818284 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:29.818559 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 10:28:29.818705 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 10:28:29.822120 26891 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:29.820840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38026 (local address 127.20.92.1:45301)
0929 10:28:29.820991 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:29.820996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:29.821080 (+ 84us) server_negotiation.cc:407] Connection header received
0929 10:28:29.821277 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:29.821280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:29.821330 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:29.821423 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:29.821981 (+ 558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38026: BlockingRecv error: recv got EOF from 127.0.0.1:38026 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260929 10:28:29.822285 20848 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677679287822-20848-0/krb5kdc/krb5cc' of type FILE
Sep 29 10:28:29 dist-test-slave-9cj5 krb5kdc[26378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:29.823415 20848 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 10:28:29 dist-test-slave-9cj5 krb5kdc[26378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:29.824442 20848 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 10:28:29 dist-test-slave-9cj5 krb5kdc[26378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.1@KRBTEST.COM
I20260929 10:28:29.834324 26891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:29.825461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38034 (local address 127.20.92.1:45301)
0929 10:28:29.825593 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:29.825599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:29.825688 (+ 89us) server_negotiation.cc:407] Connection header received
0929 10:28:29.825942 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:29.825948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:29.825995 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:29.826090 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:29.826634 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:29.827175 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:29.828211 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:29.828516 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:29.828631 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:29.831161 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:29.831165 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:29.831168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:29.831199 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:29.832647 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:29.833082 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:29.833087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:29.833089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:29.833176 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:29.833461 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:29.833465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:29.833466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:29.833637 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:29.833820 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:29.834057 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 10:28:29.834160 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 10:28:29.835379 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 26487
I20260929 10:28:29.841176 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 26621
I20260929 10:28:29.846824 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 26755
I20260929 10:28:29.852367 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 26391
2026-09-29T10:28:29Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2459 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:28:29 dist-test-slave-9cj5 krb5kdc[26904](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:28:29 dist-test-slave-9cj5 krb5kdc[26904](info): set up 2 sockets
Sep 29 10:28:29 dist-test-slave-9cj5 krb5kdc[26904](info): commencing operation
krb5kdc: starting...
W20260929 10:28:30.954984 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:28:31 dist-test-slave-9cj5 krb5kdc[26904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:28:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:28:31Z Disabled control of system clock
WARNING: no policy specified for oryx/127.20.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.62.keytab.
Entry for principal oryx/127.20.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.62.keytab.
I20260929 10:28:31.085198 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:38271
--webserver_interface=127.20.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.62.keytab
--principal=oryx/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:45279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:38271
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:31.202183 26917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:31.202538 26917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:31.202620 26917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:31.206902 26917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:28:31.207015 26917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:31.207054 26917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:31.207093 26917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:28:31.207129 26917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:28:31.213307 26917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:45279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:38271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.62.keytab
--principal=oryx/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:38271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:31.214530 26917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:31.215613 26917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:28:31.224788 26917 server_base.cc:1034] running on GCE node
W20260929 10:28:31.224781 26922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:31.224793 26925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:31.224849 26923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:31.225795 26917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:31.226958 26917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:31.228142 26917 hybrid_clock.cc:648] HybridClock initialized: now 1790677711228106 us; error 55 us; skew 500 ppm
Sep 29 10:28:31 dist-test-slave-9cj5 krb5kdc[26904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677711, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:31.231854 26917 init.cc:377] Logged in from keytab as oryx/127.20.92.62@KRBTEST.COM (short username oryx)
I20260929 10:28:31.232939 26917 webserver.cc:460] Webserver started at http://127.20.92.62:40241/ using document root <none> and password file <none>
I20260929 10:28:31.233489 26917 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:31.233551 26917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:31.233722 26917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:31.236279 26917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "47d8539835204349b4a1620d4eabe0a7"
format_stamp: "Formatted at 2026-09-29 10:28:31 on dist-test-slave-9cj5"
server_key: "8e74069ce30851e8d7a625f802eb0e21"
server_key_iv: "75625bb9336e8aee3e0cf8373552aa9d"
server_key_version: "encryptionkey@0"
I20260929 10:28:31.236888 26917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "47d8539835204349b4a1620d4eabe0a7"
format_stamp: "Formatted at 2026-09-29 10:28:31 on dist-test-slave-9cj5"
server_key: "8e74069ce30851e8d7a625f802eb0e21"
server_key_iv: "75625bb9336e8aee3e0cf8373552aa9d"
server_key_version: "encryptionkey@0"
I20260929 10:28:31.241318 26917 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260929 10:28:31.244288 26932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:31.245728 26917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:28:31.246001 26917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "47d8539835204349b4a1620d4eabe0a7"
format_stamp: "Formatted at 2026-09-29 10:28:31 on dist-test-slave-9cj5"
server_key: "8e74069ce30851e8d7a625f802eb0e21"
server_key_iv: "75625bb9336e8aee3e0cf8373552aa9d"
server_key_version: "encryptionkey@0"
I20260929 10:28:31.246134 26917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:31.265270 26917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:31.269068 26917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:31.269312 26917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:31.276938 26917 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:38271
I20260929 10:28:31.277218 26984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:38271 every 8 connection(s)
I20260929 10:28:31.278117 26917 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:28:31.281108 26985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:31.282647 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 26917
I20260929 10:28:31.282754 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:28:31.283075 20848 external_mini_cluster.cc:1428] Setting key a45e2cb6c9227bc2fd8c0fd228c1240b
I20260929 10:28:31.288511 26985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7: Bootstrap starting.
I20260929 10:28:31.291527 26985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:31.292351 26985 log.cc:824] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7: Log is configured to *not* fsync() on all Append() calls
Sep 29 10:28:31 dist-test-slave-9cj5 krb5kdc[26904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM
I20260929 10:28:31.294821 26985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7: No bootstrap required, opened a new log
I20260929 10:28:31.300081 26985 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } }
I20260929 10:28:31.300300 26988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:31.284747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39018 (local address 127.20.92.62:38271)
0929 10:28:31.285238 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:31.285256 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:31.285297 (+ 41us) server_negotiation.cc:407] Connection header received
0929 10:28:31.286075 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:31.286095 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:31.286471 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:31.286783 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:31.287866 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:31.289094 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:31.290409 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:31.290940 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:31.291081 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:31.294264 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:31.294271 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:31.294289 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:31.294323 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:31.296820 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:31.297561 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:31.297568 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:31.297574 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:31.297710 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:31.298099 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:31.298102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:31.298103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:31.298589 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:31.298773 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:31.299082 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 10:28:31.299446 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":170,"threads_started":1}
I20260929 10:28:31.300333 26985 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:31.300479 26985 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47d8539835204349b4a1620d4eabe0a7, State: Initialized, Role: FOLLOWER
I20260929 10:28:31.300992 26985 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } }
I20260929 10:28:31.301164 26985 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:28:31.301258 26985 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:28:31.301393 26985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:31.302745 26985 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } }
I20260929 10:28:31.303134 26985 leader_election.cc:302] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47d8539835204349b4a1620d4eabe0a7; no voters:
I20260929 10:28:31.303522 26985 leader_election.cc:288] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:28:31.303634 26990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:31.303936 26990 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [term 1 LEADER]: Becoming Leader. State: Replica: 47d8539835204349b4a1620d4eabe0a7, State: Running, Role: LEADER
I20260929 10:28:31.304291 26990 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } }
I20260929 10:28:31.304913 26985 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:28:31.306141 26992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47d8539835204349b4a1620d4eabe0a7. Latest consensus state: current_term: 1 leader_uuid: "47d8539835204349b4a1620d4eabe0a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } }
I20260929 10:28:31.306171 26991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47d8539835204349b4a1620d4eabe0a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } }
I20260929 10:28:31.306320 26992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:31.306325 26991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:31.306762 26999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:28:31.310189 26999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:28:31.316661 26999 catalog_manager.cc:1347] Generated new cluster ID: e20e448549d846c1b8d5f05daf6c93da
I20260929 10:28:31.316751 26999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:28:31.329043 26999 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:28:31.330092 26999 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:28:31.334877 26999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7: Generated new TSK 0
I20260929 10:28:31.335685 26999 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.20.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.1.keytab.
Entry for principal oryx/127.20.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.1.keytab.
I20260929 10:28:31.374766 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:0
--local_ip_for_outbound_sockets=127.20.92.1
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:38271
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.1.keytab
--principal=oryx/127.20.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:45279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:31.491176 27011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:31.491453 27011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:31.491497 27011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:31.495638 27011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:31.495702 27011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:31.495759 27011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:28:31.500988 27011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:45279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.1.keytab
--principal=oryx/127.20.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:38271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:31.502115 27011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:31.503150 27011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:31.516486 27017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:31.516479 27016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:31.516966 27019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:31.516978 27011 server_base.cc:1034] running on GCE node
I20260929 10:28:31.518083 27011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:31.518975 27011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:31.520215 27011 hybrid_clock.cc:648] HybridClock initialized: now 1790677711520194 us; error 40 us; skew 500 ppm
Sep 29 10:28:31 dist-test-slave-9cj5 krb5kdc[26904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677711, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:31.523339 27011 init.cc:377] Logged in from keytab as oryx/127.20.92.1@KRBTEST.COM (short username oryx)
I20260929 10:28:31.524196 27011 webserver.cc:460] Webserver started at http://127.20.92.1:38309/ using document root <none> and password file <none>
I20260929 10:28:31.524744 27011 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:31.524806 27011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:31.524974 27011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:31.527081 27011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-0/data/instance:
uuid: "b2c6034747b64db3910457be844cc69b"
format_stamp: "Formatted at 2026-09-29 10:28:31 on dist-test-slave-9cj5"
server_key: "e5a2ae9597c71c5e0c415aa6a7754b60"
server_key_iv: "1d2806ded52e1a40128d87f254d88f52"
server_key_version: "encryptionkey@0"
I20260929 10:28:31.527624 27011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance:
uuid: "b2c6034747b64db3910457be844cc69b"
format_stamp: "Formatted at 2026-09-29 10:28:31 on dist-test-slave-9cj5"
server_key: "e5a2ae9597c71c5e0c415aa6a7754b60"
server_key_iv: "1d2806ded52e1a40128d87f254d88f52"
server_key_version: "encryptionkey@0"
I20260929 10:28:31.531486 27011 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:28:31.534133 27026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:31.535605 27011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 10:28:31.535722 27011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "b2c6034747b64db3910457be844cc69b"
format_stamp: "Formatted at 2026-09-29 10:28:31 on dist-test-slave-9cj5"
server_key: "e5a2ae9597c71c5e0c415aa6a7754b60"
server_key_iv: "1d2806ded52e1a40128d87f254d88f52"
server_key_version: "encryptionkey@0"
I20260929 10:28:31.535887 27011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:31.564255 27011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:31.567685 27011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:31.567935 27011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:31.568591 27011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:31.569501 27011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:31.569551 27011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:31.569624 27011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:31.569661 27011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:31.579756 27011 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:35207
I20260929 10:28:31.579777 27139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:35207 every 8 connection(s)
I20260929 10:28:31.580826 27011 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
I20260929 10:28:31.582393 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 27011
I20260929 10:28:31.582538 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance
I20260929 10:28:31.582803 20848 external_mini_cluster.cc:1428] Setting key cf8884bfbded3674266b708c8d5f614a
WARNING: no policy specified for oryx/127.20.92.2@KRBTEST.COM; defaulting to no policy
Sep 29 10:28:31 dist-test-slave-9cj5 krb5kdc[26904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677711, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.92.2@KRBTEST.COM" created.
I20260929 10:28:31.598780 26988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:31.583198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:54911 (local address 127.20.92.62:38271)
0929 10:28:31.583359 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:31.583365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:31.584370 (+ 1005us) server_negotiation.cc:407] Connection header received
0929 10:28:31.585905 (+ 1535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:31.585908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:31.585953 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:31.586012 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:31.587833 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:31.588518 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:31.590508 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:31.590826 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:31.590931 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:31.594187 (+ 3256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:31.594191 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:31.594193 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:31.594219 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:31.595725 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:31.596489 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:31.596495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:31.596497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:31.596600 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:31.597128 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:31.597130 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:31.597132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:31.597381 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:31.597492 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:31.598339 (+ 847us) server_negotiation.cc:299] Negotiation successful
0929 10:28:31.598519 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 10:28:31.599696 27142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:31.583666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.20.92.1:54911)
0929 10:28:31.584219 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:31.584276 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:31.585641 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:31.586214 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:31.586227 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:31.586776 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:31.587540 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:31.587564 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:31.588675 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:31.588681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:31.590342 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:31.590359 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:31.591015 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:31.591020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:31.591276 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:31.592050 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:31.592079 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:31.593988 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:31.595883 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:31.595889 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:31.595904 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:31.596357 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:31.596716 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:31.596719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:31.596722 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:31.596955 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:31.597516 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:31.597527 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:31.597949 (+ 422us) client_negotiation.cc:769] Sending connection context
0929 10:28:31.598179 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 10:28:31.598623 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":142,"threads_started":1}
I20260929 10:28:31.601300 27140 heartbeater.cc:347] Connected to a master server at 127.20.92.62:38271
I20260929 10:28:31.601574 27140 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:31.602175 27140 heartbeater.cc:511] Master 127.20.92.62:38271 requested a full tablet report, sending...
I20260929 10:28:31.603785 26949 ts_manager.cc:194] Registered new tserver with Master: b2c6034747b64db3910457be844cc69b (127.20.92.1:35207)
I20260929 10:28:31.605193 26949 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.92.1@KRBTEST.COM'} at 127.20.92.1:54911
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.2.keytab.
Entry for principal oryx/127.20.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.2.keytab.
I20260929 10:28:31.618476 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:0
--local_ip_for_outbound_sockets=127.20.92.2
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:38271
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.2.keytab
--principal=oryx/127.20.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:45279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:31.747350 27145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:31.747716 27145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:31.747835 27145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:31.752571 27145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:31.752674 27145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:31.752790 27145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:28:31.758875 27145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:45279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.2.keytab
--principal=oryx/127.20.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:38271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:31.760172 27145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:31.761412 27145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:31.772004 27153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:31.772004 27151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:31.772004 27150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:31.772486 27145 server_base.cc:1034] running on GCE node
I20260929 10:28:31.772979 27145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:31.773603 27145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:31.774837 27145 hybrid_clock.cc:648] HybridClock initialized: now 1790677711774763 us; error 48 us; skew 500 ppm
Sep 29 10:28:31 dist-test-slave-9cj5 krb5kdc[26904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677711, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:31.778012 27145 init.cc:377] Logged in from keytab as oryx/127.20.92.2@KRBTEST.COM (short username oryx)
I20260929 10:28:31.778986 27145 webserver.cc:460] Webserver started at http://127.20.92.2:41087/ using document root <none> and password file <none>
I20260929 10:28:31.779575 27145 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:31.779639 27145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:31.779855 27145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:31.782040 27145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/data/instance:
uuid: "52a6fe68df7942e4bd2675a0075bac88"
format_stamp: "Formatted at 2026-09-29 10:28:31 on dist-test-slave-9cj5"
server_key: "da1cfc704082c4dc8796191daa402f9f"
server_key_iv: "93b8129d5624020378598042d1bdce60"
server_key_version: "encryptionkey@0"
I20260929 10:28:31.782609 27145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance:
uuid: "52a6fe68df7942e4bd2675a0075bac88"
format_stamp: "Formatted at 2026-09-29 10:28:31 on dist-test-slave-9cj5"
server_key: "da1cfc704082c4dc8796191daa402f9f"
server_key_iv: "93b8129d5624020378598042d1bdce60"
server_key_version: "encryptionkey@0"
I20260929 10:28:31.786559 27145 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 10:28:31.789011 27160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:31.790541 27145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 10:28:31.790704 27145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "52a6fe68df7942e4bd2675a0075bac88"
format_stamp: "Formatted at 2026-09-29 10:28:31 on dist-test-slave-9cj5"
server_key: "da1cfc704082c4dc8796191daa402f9f"
server_key_iv: "93b8129d5624020378598042d1bdce60"
server_key_version: "encryptionkey@0"
I20260929 10:28:31.790838 27145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:31.816386 27145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:31.820467 27145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:31.820770 27145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:31.821846 27145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:31.822793 27145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:31.822876 27145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:31.822954 27145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:31.823004 27145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:31.833321 27145 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:39893
I20260929 10:28:31.833348 27273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:39893 every 8 connection(s)
I20260929 10:28:31.834444 27145 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
I20260929 10:28:31.840582 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 27145
I20260929 10:28:31.840754 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance
I20260929 10:28:31.841104 20848 external_mini_cluster.cc:1428] Setting key f036d65a6aa8eef6adbc3337806a05b5
Sep 29 10:28:31 dist-test-slave-9cj5 krb5kdc[26904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677711, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM
I20260929 10:28:31.851256 26988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:31.836460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:38511 (local address 127.20.92.62:38271)
0929 10:28:31.836603 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:31.836607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:31.837605 (+ 998us) server_negotiation.cc:407] Connection header received
0929 10:28:31.839163 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:31.839167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:31.839229 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:31.839347 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:31.840714 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:31.841457 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:31.843127 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:31.843384 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:31.843475 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:31.846569 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:31.846572 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:31.846574 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:31.846600 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:31.848256 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:31.849011 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:31.849015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:31.849016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:31.849098 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:31.849640 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:31.849643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:31.849644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:31.850026 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:31.850113 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:31.850999 (+ 886us) server_negotiation.cc:299] Negotiation successful
0929 10:28:31.851124 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for oryx/127.20.92.3@KRBTEST.COM; defaulting to no policy
I20260929 10:28:31.852447 27276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:31.836893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.20.92.2:38511)
0929 10:28:31.837424 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:31.837475 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:31.838923 (+ 1448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:31.839529 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:31.839537 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:31.840025 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:31.840543 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:31.840558 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:31.841644 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:31.841646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:31.842981 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:31.842994 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:31.843560 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:31.843567 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:31.843816 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:31.844617 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:31.844640 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:31.846363 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:31.848391 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:31.848400 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:31.848423 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:31.848856 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:31.849226 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:31.849232 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:31.849235 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:31.849486 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:31.850145 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:31.850153 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:31.850586 (+ 433us) client_negotiation.cc:769] Sending connection context
0929 10:28:31.850862 (+ 276us) client_negotiation.cc:240] Negotiation successful
0929 10:28:31.851365 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":111,"threads_started":1}
I20260929 10:28:31.853849 27274 heartbeater.cc:347] Connected to a master server at 127.20.92.62:38271
I20260929 10:28:31.854192 27274 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:31.854902 27274 heartbeater.cc:511] Master 127.20.92.62:38271 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.92.3@KRBTEST.COM" created.
I20260929 10:28:31.856197 26949 ts_manager.cc:194] Registered new tserver with Master: 52a6fe68df7942e4bd2675a0075bac88 (127.20.92.2:39893)
I20260929 10:28:31.856810 26949 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.92.2@KRBTEST.COM'} at 127.20.92.2:38511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.3.keytab.
Entry for principal oryx/127.20.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.3.keytab.
I20260929 10:28:31.874413 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:0
--local_ip_for_outbound_sockets=127.20.92.3
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:38271
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.3.keytab
--principal=oryx/127.20.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:45279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:31.988350 27279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:31.988613 27279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:31.988656 27279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:31.992789 27279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:31.992857 27279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:31.992949 27279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:28:31.998598 27279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:45279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.3.keytab
--principal=oryx/127.20.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:38271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:31.999668 27279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:32.000759 27279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:32.011447 27284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:32.011421 27287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:32.011410 27285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:32.011869 27279 server_base.cc:1034] running on GCE node
I20260929 10:28:32.012271 27279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:32.012883 27279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:32.014048 27279 hybrid_clock.cc:648] HybridClock initialized: now 1790677712014036 us; error 27 us; skew 500 ppm
Sep 29 10:28:32 dist-test-slave-9cj5 krb5kdc[26904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677712, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:32.017227 27279 init.cc:377] Logged in from keytab as oryx/127.20.92.3@KRBTEST.COM (short username oryx)
I20260929 10:28:32.018317 27279 webserver.cc:460] Webserver started at http://127.20.92.3:41593/ using document root <none> and password file <none>
I20260929 10:28:32.018920 27279 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:32.019012 27279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:32.019253 27279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:32.021494 27279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/data/instance:
uuid: "77ff8f0666ff4666b205c8addfa5706f"
format_stamp: "Formatted at 2026-09-29 10:28:32 on dist-test-slave-9cj5"
server_key: "3200372c9234be73fff417316d5e73f8"
server_key_iv: "e96b27a23118490744e77699afed46dd"
server_key_version: "encryptionkey@0"
I20260929 10:28:32.022192 27279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance:
uuid: "77ff8f0666ff4666b205c8addfa5706f"
format_stamp: "Formatted at 2026-09-29 10:28:32 on dist-test-slave-9cj5"
server_key: "3200372c9234be73fff417316d5e73f8"
server_key_iv: "e96b27a23118490744e77699afed46dd"
server_key_version: "encryptionkey@0"
I20260929 10:28:32.026153 27279 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 10:28:32.028595 27294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:32.030135 27279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:28:32.030282 27279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "77ff8f0666ff4666b205c8addfa5706f"
format_stamp: "Formatted at 2026-09-29 10:28:32 on dist-test-slave-9cj5"
server_key: "3200372c9234be73fff417316d5e73f8"
server_key_iv: "e96b27a23118490744e77699afed46dd"
server_key_version: "encryptionkey@0"
I20260929 10:28:32.030398 27279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:32.058715 27279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:32.062366 27279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:32.062587 27279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:32.063232 27279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:32.064097 27279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:32.064170 27279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:32.064239 27279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:32.064282 27279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:32.075126 27279 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:40193
I20260929 10:28:32.075158 27407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:40193 every 8 connection(s)
I20260929 10:28:32.076156 27279 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:28:32.081866 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 27279
I20260929 10:28:32.082019 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance
I20260929 10:28:32.082329 20848 external_mini_cluster.cc:1428] Setting key 182a1d06b81e9459d5de3d1b477459d2
Sep 29 10:28:32 dist-test-slave-9cj5 krb5kdc[26904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677712, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM
I20260929 10:28:32.090507 26988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.078175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:37967 (local address 127.20.92.62:38271)
0929 10:28:32.078359 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:32.078363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:32.079472 (+ 1109us) server_negotiation.cc:407] Connection header received
0929 10:28:32.080421 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:32.080425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:32.080486 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:32.080589 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:32.081835 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.082533 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.083672 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.083913 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.084026 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.086827 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:32.086830 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:32.086832 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:32.086856 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:32.088265 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.088911 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.088914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.088915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.088998 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.089324 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.089328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.089330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.089514 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:32.089609 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:32.090266 (+ 657us) server_negotiation.cc:299] Negotiation successful
0929 10:28:32.090384 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 10:28:32.091338 27410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.078541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.20.92.3:37967)
0929 10:28:32.079307 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:32.079348 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:32.080184 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:32.080776 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:32.080783 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:32.081179 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:32.081661 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:32.081674 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.082655 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.082661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:32.083543 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:32.083552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.084064 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.084069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:32.084251 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.084915 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:32.084932 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:32.086646 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:32.088430 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:32.088435 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:32.088448 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:32.088806 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:32.089084 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:32.089087 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:32.089088 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:32.089230 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:32.089613 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:32.089618 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:32.090027 (+ 409us) client_negotiation.cc:769] Sending connection context
0929 10:28:32.090204 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 10:28:32.090473 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":124,"threads_started":1}
I20260929 10:28:32.092587 27408 heartbeater.cc:347] Connected to a master server at 127.20.92.62:38271
I20260929 10:28:32.092865 27408 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:32.093325 27408 heartbeater.cc:511] Master 127.20.92.62:38271 requested a full tablet report, sending...
I20260929 10:28:32.094339 26949 ts_manager.cc:194] Registered new tserver with Master: 77ff8f0666ff4666b205c8addfa5706f (127.20.92.3:40193)
I20260929 10:28:32.095094 26949 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.92.3@KRBTEST.COM'} at 127.20.92.3:37967
I20260929 10:28:32.096782 20848 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.20.92.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.92.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.92.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.61.keytab.
Entry for principal oryx/127.20.92.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.61.keytab.
I20260929 10:28:32.129506 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.61:41373
--webserver_interface=127.20.92.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.61.keytab
--principal=oryx/127.20.92.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:45279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:38271,127.20.92.61:41373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:32.263367 27413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:32.263689 27413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:32.263732 27413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:32.268025 27413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:28:32.268101 27413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:32.268121 27413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:32.268137 27413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:28:32.268153 27413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:28:32.273655 27413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:45279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:38271,127.20.92.61:41373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.61.keytab
--principal=oryx/127.20.92.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.61:41373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.92.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
W20260929 10:28:32.273968 27413 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.20.92.62:38271,127.20.92.61:41373'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 10:28:32.274991 27413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:32.276080 27413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:32.285869 27421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:32.285969 27419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:32.285915 27418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:32.286465 27413 server_base.cc:1034] running on GCE node
I20260929 10:28:32.287158 27413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:32.288602 27413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:32.289904 27413 hybrid_clock.cc:648] HybridClock initialized: now 1790677712289877 us; error 41 us; skew 500 ppm
Sep 29 10:28:32 dist-test-slave-9cj5 krb5kdc[26904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677712, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:32.292853 27413 init.cc:377] Logged in from keytab as oryx/127.20.92.61@KRBTEST.COM (short username oryx)
I20260929 10:28:32.293869 27413 webserver.cc:460] Webserver started at http://127.20.92.61:42559/ using document root <none> and password file <none>
I20260929 10:28:32.294437 27413 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:32.294528 27413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:32.294798 27413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:32.297022 27413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-1/data/instance:
uuid: "1c21be5e2e22417fa95a888fc5205dca"
format_stamp: "Formatted at 2026-09-29 10:28:32 on dist-test-slave-9cj5"
server_key: "2fd065bcc1fdd86a7d864395d26d2359"
server_key_iv: "12643758f229efe680dda910fcadaf30"
server_key_version: "encryptionkey@0"
I20260929 10:28:32.297828 27413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-1/wal/instance:
uuid: "1c21be5e2e22417fa95a888fc5205dca"
format_stamp: "Formatted at 2026-09-29 10:28:32 on dist-test-slave-9cj5"
server_key: "2fd065bcc1fdd86a7d864395d26d2359"
server_key_iv: "12643758f229efe680dda910fcadaf30"
server_key_version: "encryptionkey@0"
I20260929 10:28:32.301966 27413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 10:28:32.304667 27428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:32.306831 27413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:28:32.307009 27413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-1/wal
uuid: "1c21be5e2e22417fa95a888fc5205dca"
format_stamp: "Formatted at 2026-09-29 10:28:32 on dist-test-slave-9cj5"
server_key: "2fd065bcc1fdd86a7d864395d26d2359"
server_key_iv: "12643758f229efe680dda910fcadaf30"
server_key_version: "encryptionkey@0"
I20260929 10:28:32.307125 27413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:32.320358 27413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:32.324715 27413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:32.325088 27413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:32.333313 27413 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.61:41373
I20260929 10:28:32.333595 27480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.61:41373 every 8 connection(s)
I20260929 10:28:32.334535 27413 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-1/data/info.pb
I20260929 10:28:32.337669 27481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:32.337903 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 27413
I20260929 10:28:32.342721 27481 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 10:28:32 dist-test-slave-9cj5 krb5kdc[26904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.92.61@KRBTEST.COM
Sep 29 10:28:32 dist-test-slave-9cj5 krb5kdc[26904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677712, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.61@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM
I20260929 10:28:32.354516 27484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.339451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49134 (local address 127.20.92.61:41373)
0929 10:28:32.339947 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:32.339962 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:32.339998 (+ 36us) server_negotiation.cc:407] Connection header received
0929 10:28:32.340741 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:32.340759 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:32.341099 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:32.341459 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:32.342329 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.343387 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.344273 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.344924 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.345100 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.348381 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:32.348387 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:32.348397 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:32.348428 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:32.350754 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.351289 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.351296 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.351303 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.351446 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.351806 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.351811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.351815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.352396 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:32.352622 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:32.353007 (+ 385us) server_negotiation.cc:299] Negotiation successful
0929 10:28:32.353342 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":157,"threads_started":1}
I20260929 10:28:32.355485 27485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.344529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.0.0.1:36324)
0929 10:28:32.344937 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:32.344988 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:32.345140 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:32.345619 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:32.345626 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:32.346167 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:32.346308 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:32.346315 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.347427 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.347430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:32.348201 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:32.348211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.348666 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.348670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:32.348791 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.349375 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:32.349393 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:32.351331 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:32.353563 (+ 2232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:32.353570 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:32.353575 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:32.353885 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:32.354153 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:32.354156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:32.354158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:32.354258 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:32.354946 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:32.354950 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:32.355046 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 10:28:32.355159 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 10:28:32.355307 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":113,"threads_started":1}
I20260929 10:28:32.355577 26988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.344606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36324 (local address 127.20.92.62:38271)
0929 10:28:32.344761 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:32.344765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:32.345053 (+ 288us) server_negotiation.cc:407] Connection header received
0929 10:28:32.345322 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:32.345327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:32.345395 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:32.345497 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:32.346467 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.347246 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.348337 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.348553 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.348666 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.351486 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:32.351489 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:32.351491 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:32.351545 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:32.353461 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.353985 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.353988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.353989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.354065 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.354396 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.354400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.354401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.354846 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:32.354984 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:32.355279 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 10:28:32.355419 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 10:28:32.356990 27481 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 10:28:32 dist-test-slave-9cj5 krb5kdc[26904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677712, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.61@KRBTEST.COM for oryx/127.20.92.61@KRBTEST.COM
I20260929 10:28:32.366712 27485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.357524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:41373 (local address 127.0.0.1:49142)
0929 10:28:32.357650 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:32.357666 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:32.357743 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:32.358107 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:32.358109 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:32.358294 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:32.358393 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:32.358398 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.359662 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.359666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:32.360238 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:32.360246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.360726 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.360729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:32.360828 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.361395 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:32.361408 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:32.362923 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:32.364819 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:32.364822 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:32.364823 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:32.365015 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:32.365463 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:32.365466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:32.365467 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:32.365584 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:32.366273 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:32.366276 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:32.366365 (+ 89us) client_negotiation.cc:769] Sending connection context
0929 10:28:32.366463 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 10:28:32.366580 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 10:28:32.366844 27484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.357698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49142 (local address 127.20.92.61:41373)
0929 10:28:32.357890 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:32.357893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:32.357906 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:28:32.357936 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:32.357939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:32.357974 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:32.358051 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:32.358526 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.359104 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.360372 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.360602 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.360693 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.363067 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:32.363070 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:32.363072 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:32.363099 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:32.364606 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.365135 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.365138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.365140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.365229 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.365730 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.365733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.365734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.366128 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:32.366219 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:32.366548 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 10:28:32.366650 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260929 10:28:32.370115 27481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca: Bootstrap starting.
I20260929 10:28:32.372449 27481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:32.373150 27481 log.cc:824] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:32.375357 27481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca: No bootstrap required, opened a new log
I20260929 10:28:32.378840 27481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } }
I20260929 10:28:32.379087 27481 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:32.379124 27481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c21be5e2e22417fa95a888fc5205dca, State: Initialized, Role: FOLLOWER
I20260929 10:28:32.379519 27481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } }
I20260929 10:28:32.380002 27487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } } }
I20260929 10:28:32.380133 27487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:28:32.380825 27481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [sys.catalog]: configured and running, proceeding with master startup.
W20260929 10:28:32.386435 27498 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:28:32.386550 27498 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:28:32.389864 27413 master_runner.cc:428] Detected that this master 1c21be5e2e22417fa95a888fc5205dca is joining an existing cluster
I20260929 10:28:32.389967 27413 master_runner.cc:432] Initiating AddMaster RPC to add 127.20.92.61:41373
I20260929 10:28:32.402233 27504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.392442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.0.0.1:36336)
0929 10:28:32.392881 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:32.392897 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:32.393053 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:32.393458 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:32.393461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:32.393907 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:32.394152 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:32.394161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.395051 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.395057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:32.396511 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:32.396518 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.397074 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.397078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:32.397296 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.397713 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:32.397729 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:32.398290 (+ 561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:32.400483 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:32.400486 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:32.400488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:32.400740 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:32.401058 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:32.401061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:32.401063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:32.401150 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:32.401633 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:32.401636 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:32.401730 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 10:28:32.401869 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 10:28:32.402019 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":26,"spinlock_wait_cycles":22656,"thread_start_us":119,"threads_started":1}
I20260929 10:28:32.402251 26988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.392538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36336 (local address 127.20.92.62:38271)
0929 10:28:32.392677 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:32.392682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:32.392975 (+ 293us) server_negotiation.cc:407] Connection header received
0929 10:28:32.393182 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:32.393186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:32.393251 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:32.393341 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:32.394343 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.394936 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.396647 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.396947 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.397062 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.398467 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:32.398472 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:32.398474 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:32.398503 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:32.400355 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.400851 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.400854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.400856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.400951 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.401261 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.401265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.401267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.401491 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:32.401564 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:32.401949 (+ 385us) server_negotiation.cc:299] Negotiation successful
0929 10:28:32.402083 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 10:28:32.402261 27484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.392903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49144 (local address 127.20.92.61:41373)
0929 10:28:32.393012 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:32.393016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:32.393031 (+ 15us) server_negotiation.cc:407] Connection header received
0929 10:28:32.393467 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:32.393470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:32.393512 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:32.393586 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:32.394283 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.395104 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.396345 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.396682 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.396822 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.398448 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:32.398452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:32.398455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:32.398480 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:32.400522 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.401107 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.401110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.401112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.401184 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.401479 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.401482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.401484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.401650 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:32.401749 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:32.402033 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 10:28:32.402134 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":44}
I20260929 10:28:32.403196 27505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.392498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:41373 (local address 127.0.0.1:49144)
0929 10:28:32.392934 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:32.392948 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:32.393053 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:32.393634 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:32.393637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:32.393927 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:32.394152 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:32.394161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.395227 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.395231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:32.396218 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:32.396229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.396842 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.396846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:32.396944 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.397559 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:32.397580 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:32.398270 (+ 690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:32.400664 (+ 2394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:32.400669 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:32.400671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:32.401000 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:32.401277 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:32.401281 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:32.401283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:32.401368 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:32.401783 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:32.401786 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:32.401868 (+ 82us) client_negotiation.cc:769] Sending connection context
0929 10:28:32.402891 (+ 1023us) client_negotiation.cc:240] Negotiation successful
0929 10:28:32.403023 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":49,"spinlock_wait_cycles":2176,"thread_start_us":95,"threads_started":1}
Sep 29 10:28:32 dist-test-slave-9cj5 krb5kdc[26904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677711, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.62@KRBTEST.COM for oryx/127.20.92.61@KRBTEST.COM
WARNING: no policy specified for oryx/127.20.92.60@KRBTEST.COM; defaulting to no policy
I20260929 10:28:32.421324 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.407761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:41373 (local address 127.0.0.1:49160)
0929 10:28:32.408460 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:32.408519 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:32.408658 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:32.408990 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:32.408996 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:32.409387 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:32.409504 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:32.409510 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.410467 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.410474 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:32.411899 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:32.411912 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.412572 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.412576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:32.412707 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.413393 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:32.413410 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:32.415409 (+ 1999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:32.418416 (+ 3007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:32.418432 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:32.418440 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:32.419303 (+ 863us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:32.419697 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:32.419700 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:32.419703 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:32.419819 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:32.420521 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:32.420530 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:32.420684 (+ 154us) client_negotiation.cc:769] Sending connection context
0929 10:28:32.420878 (+ 194us) client_negotiation.cc:240] Negotiation successful
0929 10:28:32.421125 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":119,"threads_started":1}
I20260929 10:28:32.421345 27484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.408585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49160 (local address 127.20.92.61:41373)
0929 10:28:32.408709 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:32.408716 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:32.408731 (+ 15us) server_negotiation.cc:407] Connection header received
0929 10:28:32.408797 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:32.408800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:32.408855 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:32.408947 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:32.409641 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.410303 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.412037 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.412432 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.412557 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.415612 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:32.415616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:32.415619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:32.415652 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:32.418162 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.419492 (+ 1330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.419496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.419497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.419587 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.419974 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.419977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.419978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.420232 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:32.420343 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:32.421102 (+ 759us) server_negotiation.cc:299] Negotiation successful
0929 10:28:32.421230 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 10:28:32.423027 26945 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.20.92.61:41373: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca"
member_type: NON_VOTER
last_known_addr {
host: "127.20.92.61"
port: 41373
}
attrs {
promote: true
}
}
dest_uuid: "47d8539835204349b4a1620d4eabe0a7"
cas_config_opid_index: -1
I20260929 10:28:32.423784 26945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: NON_VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.92.60@KRBTEST.COM" created.
I20260929 10:28:32.425994 27510 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1c21be5e2e22417fa95a888fc5205dca (127.20.92.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: NON_VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: true } } }
I20260929 10:28:32.426973 27453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:32.428277 27510 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.20.92.61:41373
I20260929 10:28:32.428627 27453 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [term 1 FOLLOWER]: Refusing update from remote peer 47d8539835204349b4a1620d4eabe0a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 10:28:32.428705 27512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "47d8539835204349b4a1620d4eabe0a7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: NON_VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: true } } }
I20260929 10:28:32.428815 27413 master.cc:537] Master@127.20.92.61:41373 shutting down...
I20260929 10:28:32.428828 27512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:32.429489 27512 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: NON_VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 10:28:32.432176 26935 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260929 10:28:32.433585 26935 consensus_peers.cc:487] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 -> Peer 1c21be5e2e22417fa95a888fc5205dca (127.20.92.61:41373): Couldn't send request to peer 1c21be5e2e22417fa95a888fc5205dca. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 10:28:32.434113 27413 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:28:32.434456 27413 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:32.434566 27413 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca: stopping tablet replica
I20260929 10:28:32.440024 27413 master.cc:552] Master@127.20.92.61:41373 shutdown complete.
I20260929 10:28:32.440089 27413 master_runner.cc:305] Clearing existing system tablet
I20260929 10:28:32.440352 27413 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:32.440407 27413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:32.441802 27413 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:28:32.443163 27516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:32.443480 27413 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:32.443610 27413 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:28:32.443645 27413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-1/wal
uuid: "1c21be5e2e22417fa95a888fc5205dca"
format_stamp: "Formatted at 2026-09-29 10:28:32 on dist-test-slave-9cj5"
server_key: "2fd065bcc1fdd86a7d864395d26d2359"
server_key_iv: "12643758f229efe680dda910fcadaf30"
server_key_version: "encryptionkey@0"
I20260929 10:28:32.446693 27413 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:32.450106 27413 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 10:28:32.450217 27413 log.cc:1196] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 10:28:32.450495 27413 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca: Deleting consensus metadata
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.92.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.60.keytab.
Entry for principal oryx/127.20.92.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.60.keytab.
I20260929 10:28:32.450927 27413 master_runner.cc:315] Copying system tablet from 127.20.92.62:38271
I20260929 10:28:32.451793 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.60:46243
--webserver_interface=127.20.92.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.60.keytab
--principal=oryx/127.20.92.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:45279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:38271,127.20.92.61:41373,127.20.92.60:46243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:28:32.452186 27413 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca: tablet copy: Beginning tablet copy session from remote peer at address 127.20.92.62:38271
I20260929 10:28:32.463114 26988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.453599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36340 (local address 127.20.92.62:38271)
0929 10:28:32.453734 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:32.453739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:32.454803 (+ 1064us) server_negotiation.cc:407] Connection header received
0929 10:28:32.455051 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:32.455055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:32.455114 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:32.455206 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:32.455673 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.456335 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.457628 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.457973 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.458094 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.459432 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:32.459436 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:32.459439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:32.459474 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:32.461336 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.461818 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.461823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.461826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.461923 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.462212 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.462215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.462217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.462426 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:32.462539 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:32.462825 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 10:28:32.462943 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 10:28:32.463129 27523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.453499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.0.0.1:36340)
0929 10:28:32.454666 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:32.454680 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:32.454929 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:32.455223 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:32.455226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:32.455453 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:32.455546 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:32.455552 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.456573 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.456577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:32.457481 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:32.457491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.458099 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.458103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:32.458188 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.458884 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:32.458897 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:32.459273 (+ 376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:32.461464 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:32.461468 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:32.461471 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:32.461684 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:32.462020 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:32.462025 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:32.462027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:32.462110 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:32.462525 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:32.462528 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:32.462636 (+ 108us) client_negotiation.cc:769] Sending connection context
0929 10:28:32.462749 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 10:28:32.462927 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1090,"thread_start_us":101,"threads_started":1}
I20260929 10:28:32.464089 26969 tablet_copy_service.cc:140] P 47d8539835204349b4a1620d4eabe0a7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1c21be5e2e22417fa95a888fc5205dca ({username='oryx', principal='oryx/127.20.92.61@KRBTEST.COM'} at 127.0.0.1:36340)
I20260929 10:28:32.464282 26969 tablet_copy_service.cc:161] P 47d8539835204349b4a1620d4eabe0a7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1c21be5e2e22417fa95a888fc5205dca at {username='oryx', principal='oryx/127.20.92.61@KRBTEST.COM'} at 127.0.0.1:36340: session id = 1c21be5e2e22417fa95a888fc5205dca-00000000000000000000000000000000
I20260929 10:28:32.465727 26969 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 10:28:32.467422 27413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 10:28:32.469419 27413 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca: tablet copy: Starting download of 0 data blocks...
I20260929 10:28:32.469643 27413 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca: tablet copy: Starting download of 1 WAL segments...
I20260929 10:28:32.471477 27413 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 10:28:32.473230 26969 tablet_copy_service.cc:342] P 47d8539835204349b4a1620d4eabe0a7: Request end of tablet copy session 1c21be5e2e22417fa95a888fc5205dca-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.20.92.61@KRBTEST.COM'} at 127.0.0.1:36340
I20260929 10:28:32.473376 26969 tablet_copy_service.cc:433] P 47d8539835204349b4a1620d4eabe0a7: ending tablet copy session 1c21be5e2e22417fa95a888fc5205dca-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1c21be5e2e22417fa95a888fc5205dca
I20260929 10:28:32.480469 27413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:32.482661 27524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:32.482671 27413 server_base.cc:1034] running on GCE node
W20260929 10:28:32.482942 27527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:32.483117 27525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:32.483346 27413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:32.483683 27413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:32.484867 27413 hybrid_clock.cc:648] HybridClock initialized: now 1790677712484854 us; error 44 us; skew 500 ppm
Sep 29 10:28:32 dist-test-slave-9cj5 krb5kdc[26904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677712, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:32.486523 27413 init.cc:377] Logged in from keytab as oryx/127.20.92.61@KRBTEST.COM (short username oryx)
I20260929 10:28:32.487041 27413 webserver.cc:460] Webserver started at http://127.20.92.61:39573/ using document root <none> and password file <none>
I20260929 10:28:32.487263 27413 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:32.487318 27413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:32.488445 27413 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:28:32.489401 27534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:32.489647 27413 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:28:32.489764 27413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-1/wal
uuid: "1c21be5e2e22417fa95a888fc5205dca"
format_stamp: "Formatted at 2026-09-29 10:28:32 on dist-test-slave-9cj5"
server_key: "2fd065bcc1fdd86a7d864395d26d2359"
server_key_iv: "12643758f229efe680dda910fcadaf30"
server_key_version: "encryptionkey@0"
I20260929 10:28:32.489859 27413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:32.500072 27413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:32.502233 27413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:32.507282 27413 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.61:41373
I20260929 10:28:32.507894 27413 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-1/data/info.pb
I20260929 10:28:32.508301 27586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.61:41373 every 8 connection(s)
I20260929 10:28:32.509621 27587 sys_catalog.cc:263] Verifying existing consensus state
I20260929 10:28:32.510502 27587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca: Bootstrap starting.
I20260929 10:28:32.519148 27587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:28:32.519622 27587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca: Bootstrap complete.
I20260929 10:28:32.520638 27587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: NON_VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: true } }
I20260929 10:28:32.520898 27587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c21be5e2e22417fa95a888fc5205dca, State: Initialized, Role: LEARNER
I20260929 10:28:32.521031 27587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: NON_VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: true } }
I20260929 10:28:32.521426 27590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: NON_VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: true } } }
I20260929 10:28:32.521605 27587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:28:32.521682 27590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [sys.catalog]: This master's current role is: LEARNER
I20260929 10:28:32.525266 27601 catalog_manager.cc:1259] Loaded cluster ID: e20e448549d846c1b8d5f05daf6c93da
I20260929 10:28:32.525488 27601 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca: loading cluster ID for follower catalog manager: success
I20260929 10:28:32.527213 27601 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca: acquiring CA information for follower catalog manager: success
I20260929 10:28:32.528650 27601 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 10:28:32.608108 27140 heartbeater.cc:503] Master 127.20.92.62:38271 was elected leader, sending a full tablet report...
W20260929 10:28:32.623771 27522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:32.624153 27522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:32.624205 27522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:32.628921 27522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:28:32.629010 27522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:32.629032 27522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:32.629051 27522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:28:32.629069 27522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:28:32.636075 27522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:45279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:38271,127.20.92.61:41373,127.20.92.60:46243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.60.keytab
--principal=oryx/127.20.92.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.60:46243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.92.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:32.637248 27522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:32.638422 27522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:32.648553 27607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:32.648553 27606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:32.648706 27609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:32.649134 27522 server_base.cc:1034] running on GCE node
I20260929 10:28:32.649816 27522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:32.651130 27522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:32.652360 27522 hybrid_clock.cc:648] HybridClock initialized: now 1790677712652250 us; error 124 us; skew 500 ppm
Sep 29 10:28:32 dist-test-slave-9cj5 krb5kdc[26904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677712, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:32.655903 27522 init.cc:377] Logged in from keytab as oryx/127.20.92.60@KRBTEST.COM (short username oryx)
I20260929 10:28:32.657230 27522 webserver.cc:460] Webserver started at http://127.20.92.60:34271/ using document root <none> and password file <none>
I20260929 10:28:32.657931 27522 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:32.658008 27522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:32.658258 27522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:32.660475 27522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-2/data/instance:
uuid: "47387bc665724e9792490842b4ffc9ae"
format_stamp: "Formatted at 2026-09-29 10:28:32 on dist-test-slave-9cj5"
server_key: "147b8a448b0df85badecede698d7f649"
server_key_iv: "6f8b41ac6de212345a64602224cc37da"
server_key_version: "encryptionkey@0"
I20260929 10:28:32.661072 27522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-2/wal/instance:
uuid: "47387bc665724e9792490842b4ffc9ae"
format_stamp: "Formatted at 2026-09-29 10:28:32 on dist-test-slave-9cj5"
server_key: "147b8a448b0df85badecede698d7f649"
server_key_iv: "6f8b41ac6de212345a64602224cc37da"
server_key_version: "encryptionkey@0"
I20260929 10:28:32.665542 27522 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 10:28:32.668614 27616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:32.670378 27522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:28:32.670557 27522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-2/wal
uuid: "47387bc665724e9792490842b4ffc9ae"
format_stamp: "Formatted at 2026-09-29 10:28:32 on dist-test-slave-9cj5"
server_key: "147b8a448b0df85badecede698d7f649"
server_key_iv: "6f8b41ac6de212345a64602224cc37da"
server_key_version: "encryptionkey@0"
I20260929 10:28:32.670696 27522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:32.686276 27522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:32.689524 27522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:32.689788 27522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:32.697187 27522 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.60:46243
I20260929 10:28:32.697525 27668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.60:46243 every 8 connection(s)
I20260929 10:28:32.698300 27522 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-2/data/info.pb
I20260929 10:28:32.701246 27669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:32.701426 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 27522
I20260929 10:28:32.706881 27669 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 10:28:32 dist-test-slave-9cj5 krb5kdc[26904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.92.60@KRBTEST.COM
Sep 29 10:28:32 dist-test-slave-9cj5 krb5kdc[26904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677712, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.60@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM
I20260929 10:28:32.718477 26988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.708872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36346 (local address 127.20.92.62:38271)
0929 10:28:32.709048 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:32.709054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:32.709478 (+ 424us) server_negotiation.cc:407] Connection header received
0929 10:28:32.709586 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:32.709589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:32.709641 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:32.709734 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:32.710742 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.711265 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.712271 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.712555 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.712669 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.715022 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:32.715026 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:32.715028 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:32.715051 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:32.716644 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.717164 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.717167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.717169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.717257 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.717554 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.717559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.717560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.717811 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:32.717917 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:32.718225 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 10:28:32.718341 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 10:28:32.718660 27672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.708810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.0.0.1:36346)
0929 10:28:32.709318 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:32.709368 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:32.709491 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:32.710015 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:32.710020 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:32.710389 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:32.710514 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:32.710520 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.711418 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.711421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:32.712137 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:32.712146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.712708 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.712712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:32.712825 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.713325 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:32.713336 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:32.714873 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:32.716758 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:32.716767 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:32.716773 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:32.717069 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:32.717377 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:32.717379 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:32.717381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:32.717459 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:32.717929 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:32.717934 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:32.718033 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 10:28:32.718149 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 10:28:32.718297 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":179,"threads_started":1}
I20260929 10:28:32.718657 27671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.703233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47102 (local address 127.20.92.60:46243)
0929 10:28:32.703816 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:32.703833 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:32.703923 (+ 90us) server_negotiation.cc:407] Connection header received
0929 10:28:32.704713 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:32.704732 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:32.705326 (+ 594us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:32.705905 (+ 579us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:32.706910 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.708090 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.709080 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.709917 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.710092 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.712852 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:32.712859 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:32.712871 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:32.712907 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:32.714833 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.715275 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.715282 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.715288 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.715399 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.715692 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.715695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.715697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.716216 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:32.717060 (+ 844us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:32.717251 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 10:28:32.717584 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":162,"threads_started":1}
I20260929 10:28:32.720024 27669 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 10:28:32 dist-test-slave-9cj5 krb5kdc[26904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677712, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.60@KRBTEST.COM for oryx/127.20.92.61@KRBTEST.COM
I20260929 10:28:32.729712 27672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.720576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:41373 (local address 127.0.0.1:49168)
0929 10:28:32.720785 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:32.720808 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:32.720902 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:32.721424 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:32.721428 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:32.721628 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:32.721729 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:32.721735 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.722849 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.722852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:32.723530 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:32.723539 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.724035 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.724040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:32.724146 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.724683 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:32.724698 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:32.726045 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:32.728172 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:32.728175 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:32.728177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:32.728420 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:32.728690 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:32.728693 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:32.728694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:32.728776 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:32.729218 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:32.729221 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:32.729335 (+ 114us) client_negotiation.cc:769] Sending connection context
0929 10:28:32.729444 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 10:28:32.729574 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260929 10:28:32.729907 27673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.720868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49168 (local address 127.20.92.61:41373)
0929 10:28:32.721206 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:32.721210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:32.721222 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:28:32.721259 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:32.721261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:32.721302 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:32.721409 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:32.721917 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.722723 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.723651 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.723914 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.724038 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.726179 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:32.726183 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:32.726185 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:32.726209 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:32.728039 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.728516 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.728518 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.728520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.728601 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.728904 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.728907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.728908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.729100 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:32.729184 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:32.729527 (+ 343us) server_negotiation.cc:299] Negotiation successful
0929 10:28:32.729639 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":106,"threads_started":1}
I20260929 10:28:32.730664 27669 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.92.60" port: 46243 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 10:28:32 dist-test-slave-9cj5 krb5kdc[26904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677712, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.60@KRBTEST.COM for oryx/127.20.92.60@KRBTEST.COM
I20260929 10:28:32.740399 27672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.731320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:46243 (local address 127.0.0.1:47108)
0929 10:28:32.731459 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:32.731475 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:32.731555 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:32.731882 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:32.731885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:32.732034 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:32.732113 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:32.732118 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.733430 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.733434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:32.734052 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:32.734061 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.734862 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.734866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:32.735044 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.735659 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:32.735674 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:32.737060 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:32.738926 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:32.738929 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:32.738931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:32.739126 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:32.739404 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:32.739406 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:32.739408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:32.739483 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:32.739944 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:32.739947 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:32.740027 (+ 80us) client_negotiation.cc:769] Sending connection context
0929 10:28:32.740117 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 10:28:32.740224 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":5,"spinlock_wait_cycles":7040}
I20260929 10:28:32.740417 27671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.731535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47108 (local address 127.20.92.60:46243)
0929 10:28:32.731646 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:32.731649 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:32.731661 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:28:32.731689 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:32.731691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:32.731722 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:32.731795 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:32.732250 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.732876 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.734187 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.734704 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.734851 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.737224 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:32.737228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:32.737230 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:32.737252 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:32.738767 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.739215 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.739218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.739220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.739294 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.739565 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.739568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.739569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.739742 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:32.739930 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:32.740183 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 10:28:32.740281 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":165,"server-negotiator.queue_time_us":47}
I20260929 10:28:32.743904 27669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae: Bootstrap starting.
I20260929 10:28:32.746495 27669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:32.747217 27669 log.cc:824] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:32.749133 27669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae: No bootstrap required, opened a new log
I20260929 10:28:32.752516 27669 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 46243 } }
I20260929 10:28:32.752727 27669 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:32.752769 27669 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47387bc665724e9792490842b4ffc9ae, State: Initialized, Role: FOLLOWER
I20260929 10:28:32.753245 27669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 46243 } }
I20260929 10:28:32.753983 27675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 46243 } } }
I20260929 10:28:32.754101 27675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:28:32.754683 27669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [sys.catalog]: configured and running, proceeding with master startup.
W20260929 10:28:32.759610 27686 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:28:32.759711 27686 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:28:32.763720 27522 master_runner.cc:428] Detected that this master 47387bc665724e9792490842b4ffc9ae is joining an existing cluster
I20260929 10:28:32.763799 27522 master_runner.cc:432] Initiating AddMaster RPC to add 127.20.92.60:46243
I20260929 10:28:32.772708 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 27011
I20260929 10:28:32.780782 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:35207
--local_ip_for_outbound_sockets=127.20.92.1
--tserver_master_addrs=127.20.92.62:38271
--webserver_port=38309
--webserver_interface=127.20.92.1
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.1.keytab
--principal=oryx/127.20.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:45279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.92.62:38271,127.20.92.61:41373
--tserver_master_addrs=127.20.92.62:38271,127.20.92.61:41373,127.20.92.60:46243 with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:28:32.795732 27694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.766600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:46243 (local address 127.0.0.1:47118)
0929 10:28:32.766890 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:32.766903 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:32.767019 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:32.767312 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:32.767315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:32.768809 (+ 1494us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:32.769229 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:32.769240 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.770157 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.770159 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:32.771215 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:32.771225 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.771790 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.771793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:32.771878 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.772339 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:32.772358 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:32.772843 (+ 485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:32.775585 (+ 2742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:32.775591 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:32.775596 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:32.794239 (+ 18643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:32.794617 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:32.794621 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:32.794624 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:32.794722 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:32.795175 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:32.795178 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:32.795266 (+ 88us) client_negotiation.cc:769] Sending connection context
0929 10:28:32.795364 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 10:28:32.795514 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":24,"thread_start_us":74,"threads_started":1}
I20260929 10:28:32.796460 27671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.766424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47118 (local address 127.20.92.60:46243)
0929 10:28:32.766527 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:32.766531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:32.766965 (+ 434us) server_negotiation.cc:407] Connection header received
0929 10:28:32.767106 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:32.767111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:32.767146 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:32.767191 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:32.769370 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.770065 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.771349 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.771614 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.771702 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.772958 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:32.772962 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:32.772965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:32.772989 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:32.775475 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.794400 (+ 18925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.794404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.794406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.794506 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.794816 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.794819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.794821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.795040 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:32.795133 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:32.796066 (+ 933us) server_negotiation.cc:299] Negotiation successful
0929 10:28:32.796301 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 10:28:32.799487 27692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.765992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:41373 (local address 127.0.0.1:49176)
0929 10:28:32.766781 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:32.766795 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:32.766923 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:32.767275 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:32.767278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:32.767504 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:32.767595 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:32.767600 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.768331 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.768336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:32.769922 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:32.769931 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.770643 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.770646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:32.770750 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.794438 (+ 23688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:32.794457 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:32.794944 (+ 487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:32.797711 (+ 2767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:32.797714 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:32.797717 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:32.798171 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:32.798469 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:32.798472 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:32.798473 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:32.798552 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:32.798987 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:32.798990 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:32.799078 (+ 88us) client_negotiation.cc:769] Sending connection context
0929 10:28:32.799173 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 10:28:32.799312 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":568,"spinlock_wait_cycles":4864,"thread_start_us":99,"threads_started":1}
I20260929 10:28:32.799973 26988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.766064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36352 (local address 127.20.92.62:38271)
0929 10:28:32.766197 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:32.766201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:32.766454 (+ 253us) server_negotiation.cc:407] Connection header received
0929 10:28:32.766675 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:32.766679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:32.766736 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:32.766828 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:32.767488 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.768506 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.769705 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.770100 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.770220 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.771910 (+ 1690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:32.771914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:32.771917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:32.771944 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:32.774182 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.798450 (+ 24268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.798455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.798457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.798578 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.798913 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.798916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.798918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.799205 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:32.799289 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:32.799672 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 10:28:32.799810 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 10:28:32.800310 27673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.766394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49176 (local address 127.20.92.61:41373)
0929 10:28:32.766522 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:32.766526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:32.766862 (+ 336us) server_negotiation.cc:407] Connection header received
0929 10:28:32.767025 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:32.767031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:32.767084 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:32.767136 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:32.767705 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.768216 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.770040 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.770416 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.770497 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.795168 (+ 24671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:32.795172 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:32.795175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:32.795206 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:32.797576 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.798285 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.798289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.798291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.798372 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.798641 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.798644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.798646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.798863 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:32.798946 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:32.800027 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0929 10:28:32.800164 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 10:28:32.800697 27693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.765958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.0.0.1:36352)
0929 10:28:32.766373 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:32.766387 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:32.766529 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:32.766958 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:32.766964 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:32.767207 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:32.767365 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:32.767389 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.768631 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.768634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:32.769577 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:32.769585 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.770376 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.770380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:32.770508 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.771052 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:32.771067 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:32.771743 (+ 676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:32.774620 (+ 2877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:32.774623 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:32.774625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:32.798294 (+ 23669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:32.798707 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:32.798710 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:32.798713 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:32.798802 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:32.799373 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:32.799376 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:32.799474 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 10:28:32.800181 (+ 707us) client_negotiation.cc:240] Negotiation successful
0929 10:28:32.800510 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":2,"thread_start_us":104,"threads_started":1}
Sep 29 10:28:32 dist-test-slave-9cj5 krb5kdc[26904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677711, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.62@KRBTEST.COM for oryx/127.20.92.60@KRBTEST.COM
I20260929 10:28:32.814705 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.804687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:46243 (local address 127.0.0.1:47124)
0929 10:28:32.804861 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:32.804876 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:32.804959 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:32.805275 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:32.805278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:32.805475 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:32.805582 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:32.805588 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.806694 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.806698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:32.807564 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:32.807575 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.808196 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.808200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:32.808292 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.808836 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:32.808851 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:32.810544 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:32.812898 (+ 2354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:32.812902 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:32.812904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:32.813194 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:32.813509 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:32.813513 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:32.813515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:32.813605 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:32.814116 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:32.814119 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:32.814234 (+ 115us) client_negotiation.cc:769] Sending connection context
0929 10:28:32.814342 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 10:28:32.814508 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 10:28:32.815372 27671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.804761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47124 (local address 127.20.92.60:46243)
0929 10:28:32.804869 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:32.804874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:32.804920 (+ 46us) server_negotiation.cc:407] Connection header received
0929 10:28:32.805087 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:32.805091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:32.805145 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:32.805247 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:32.805700 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.806578 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.807685 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.808052 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.808190 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.810721 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:32.810726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:32.810729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:32.810756 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:32.812754 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.813314 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.813318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.813321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.813405 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.813709 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.813713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.813715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.813972 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:32.814068 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:32.815068 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0929 10:28:32.815205 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 10:28:32.816367 26945 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.20.92.60:46243: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "47387bc665724e9792490842b4ffc9ae"
member_type: NON_VOTER
last_known_addr {
host: "127.20.92.60"
port: 46243
}
attrs {
promote: true
}
}
dest_uuid: "47d8539835204349b4a1620d4eabe0a7"
cas_config_opid_index: 5
I20260929 10:28:32.817006 26945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: NON_VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: true } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: NON_VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: true } }
I20260929 10:28:32.818416 27509 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 47387bc665724e9792490842b4ffc9ae (127.20.92.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: NON_VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: true } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: NON_VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: true } } }
I20260929 10:28:32.819361 27643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:32.820075 27509 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.20.92.60:46243
I20260929 10:28:32.820264 27509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "47d8539835204349b4a1620d4eabe0a7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: NON_VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: true } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: NON_VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: true } } }
I20260929 10:28:32.820372 27509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:32.820811 27643 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [term 1 FOLLOWER]: Refusing update from remote peer 47d8539835204349b4a1620d4eabe0a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 10:28:32.821185 27522 master.cc:537] Master@127.20.92.60:46243 shutting down...
I20260929 10:28:32.821779 27512 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: NON_VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 10:28:32.824566 26934 consensus_peers.cc:487] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 -> Peer 47387bc665724e9792490842b4ffc9ae (127.20.92.60:46243): Couldn't send request to peer 47387bc665724e9792490842b4ffc9ae. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 10:28:32.827418 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.818369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:41373 (local address 127.0.0.1:49178)
0929 10:28:32.822695 (+ 4326us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:32.822711 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:32.822810 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:32.823242 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:32.823245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:32.823263 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:32.823370 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:32.823376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.824482 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.824486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:32.825851 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:32.825863 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.826948 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.826952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:32.827045 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.827104 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 10:28:32.827181 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 10:28:32.827258 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4215}
I20260929 10:28:32.827888 27673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.818643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49178 (local address 127.20.92.61:41373)
0929 10:28:32.822557 (+ 3914us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:32.822562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:32.822896 (+ 334us) server_negotiation.cc:407] Connection header received
0929 10:28:32.822951 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:32.822954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:32.823013 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:32.823094 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:32.823529 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.824349 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.825994 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.826644 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.826747 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.826883 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:32.827695 (+ 812us) server_negotiation.cc:299] Negotiation successful
0929 10:28:32.827776 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3810}
I20260929 10:28:32.828724 27554 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [term 1 LEARNER]: Refusing update from remote peer 47d8539835204349b4a1620d4eabe0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 10:28:32.828960 27522 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:28:32.829264 27522 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:32.829209 27512 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: NON_VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 10:28:32.829365 27522 tablet_replica.cc:330] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae: stopping tablet replica
I20260929 10:28:32.831072 27554 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 47387bc665724e9792490842b4ffc9ae (127.20.92.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: NON_VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: true } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: NON_VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: true } } }
I20260929 10:28:32.833207 27697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47d8539835204349b4a1620d4eabe0a7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: NON_VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: true } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: NON_VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: true } } }
I20260929 10:28:32.833333 27697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [sys.catalog]: This master's current role is: LEARNER
I20260929 10:28:32.833437 27512 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7: attempting to promote NON_VOTER 1c21be5e2e22417fa95a888fc5205dca to VOTER
I20260929 10:28:32.833930 27512 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: NON_VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: true } }
I20260929 10:28:32.835166 27561 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [term 1 LEARNER]: Refusing update from remote peer 47d8539835204349b4a1620d4eabe0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 10:28:32.835494 27590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47d8539835204349b4a1620d4eabe0a7. Latest consensus state: current_term: 1 leader_uuid: "47d8539835204349b4a1620d4eabe0a7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: NON_VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: true } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: NON_VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: true } } }
I20260929 10:28:32.835605 27590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [sys.catalog]: This master's current role is: LEARNER
I20260929 10:28:32.835670 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.834813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:46243 (local address 127.0.0.1:47130)
0929 10:28:32.835496 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:32.835584 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":602}
I20260929 10:28:32.835671 27510 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 10:28:32.836468 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.836209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:46243 (local address 127.0.0.1:47138)
0929 10:28:32.836363 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:32.836411 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260929 10:28:32.836594 26934 consensus_peers.cc:487] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 -> Peer 47387bc665724e9792490842b4ffc9ae (127.20.92.60:46243): Couldn't send request to peer 47387bc665724e9792490842b4ffc9ae. Status: Network error: Client connection negotiation failed: client connection to 127.20.92.60:46243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 10:28:32.837597 27509 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1c21be5e2e22417fa95a888fc5205dca (127.20.92.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: NON_VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: true } } }
I20260929 10:28:32.839054 27509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "47d8539835204349b4a1620d4eabe0a7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: NON_VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: true } } }
I20260929 10:28:32.839154 27509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:32.839262 27509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47d8539835204349b4a1620d4eabe0a7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: NON_VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: true } } }
I20260929 10:28:32.839339 27509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:32.839617 27561 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1c21be5e2e22417fa95a888fc5205dca (127.20.92.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: NON_VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: true } } }
I20260929 10:28:32.841158 27590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47d8539835204349b4a1620d4eabe0a7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: NON_VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: true } } }
I20260929 10:28:32.841291 27590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:28:32.846436 27522 master.cc:552] Master@127.20.92.60:46243 shutdown complete.
I20260929 10:28:32.846499 27522 master_runner.cc:305] Clearing existing system tablet
I20260929 10:28:32.846740 27522 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:32.846797 27522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:32.848197 27522 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 10:28:32.849210 27699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:32.849571 27522 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:32.849699 27522 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:28:32.849786 27522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-2/wal
uuid: "47387bc665724e9792490842b4ffc9ae"
format_stamp: "Formatted at 2026-09-29 10:28:32 on dist-test-slave-9cj5"
server_key: "147b8a448b0df85badecede698d7f649"
server_key_iv: "6f8b41ac6de212345a64602224cc37da"
server_key_version: "encryptionkey@0"
I20260929 10:28:32.852046 27522 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:32.854691 27522 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 10:28:32.854790 27522 log.cc:1196] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 10:28:32.855185 27522 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae: Deleting consensus metadata
I20260929 10:28:32.855469 27522 master_runner.cc:315] Copying system tablet from 127.20.92.62:38271
I20260929 10:28:32.856861 27522 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae: tablet copy: Beginning tablet copy session from remote peer at address 127.20.92.62:38271
I20260929 10:28:32.859428 27274 heartbeater.cc:503] Master 127.20.92.62:38271 was elected leader, sending a full tablet report...
I20260929 10:28:32.869865 27705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.860168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.0.0.1:36360)
0929 10:28:32.860540 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:32.860555 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:32.860668 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:32.861229 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:32.861234 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:32.861480 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:32.861965 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:32.861977 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.863085 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.863089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:32.864053 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:32.864065 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.864690 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.864694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:32.864790 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.865400 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:32.865415 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:32.865944 (+ 529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:32.868082 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:32.868087 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:32.868090 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:32.868361 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:32.868685 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:32.868691 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:32.868693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:32.868792 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:32.869246 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:32.869249 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:32.869355 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 10:28:32.869467 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 10:28:32.869634 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":112,"threads_started":1}
I20260929 10:28:32.869891 26988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:32.860319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36360 (local address 127.20.92.62:38271)
0929 10:28:32.860840 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:32.860844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:32.860875 (+ 31us) server_negotiation.cc:407] Connection header received
0929 10:28:32.860915 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:32.860918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:32.860962 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:32.861031 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:32.862111 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.862942 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.864192 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:32.864576 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:32.864695 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:32.866096 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:32.866100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:32.866103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:32.866132 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:32.867942 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.868478 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.868481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.868483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.868577 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:32.868906 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:32.868910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:32.868912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:32.869141 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:32.869240 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:32.869556 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 10:28:32.869680 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440}
I20260929 10:28:32.870591 26969 tablet_copy_service.cc:140] P 47d8539835204349b4a1620d4eabe0a7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 47387bc665724e9792490842b4ffc9ae ({username='oryx', principal='oryx/127.20.92.60@KRBTEST.COM'} at 127.0.0.1:36360)
I20260929 10:28:32.870688 26969 tablet_copy_service.cc:161] P 47d8539835204349b4a1620d4eabe0a7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 47387bc665724e9792490842b4ffc9ae at {username='oryx', principal='oryx/127.20.92.60@KRBTEST.COM'} at 127.0.0.1:36360: session id = 47387bc665724e9792490842b4ffc9ae-00000000000000000000000000000000
I20260929 10:28:32.872076 26969 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 10:28:32.873543 27522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 10:28:32.876037 27522 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae: tablet copy: Starting download of 0 data blocks...
I20260929 10:28:32.876227 27522 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae: tablet copy: Starting download of 1 WAL segments...
I20260929 10:28:32.877707 27522 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 10:28:32.879635 26969 tablet_copy_service.cc:342] P 47d8539835204349b4a1620d4eabe0a7: Request end of tablet copy session 47387bc665724e9792490842b4ffc9ae-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.20.92.60@KRBTEST.COM'} at 127.0.0.1:36360
I20260929 10:28:32.879730 26969 tablet_copy_service.cc:433] P 47d8539835204349b4a1620d4eabe0a7: ending tablet copy session 47387bc665724e9792490842b4ffc9ae-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 47387bc665724e9792490842b4ffc9ae
I20260929 10:28:32.888393 27522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:32.890324 27706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:32.890340 27707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:32.890781 27709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:32.891094 27522 server_base.cc:1034] running on GCE node
I20260929 10:28:32.891274 27522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:32.891587 27522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:32.892848 27522 hybrid_clock.cc:648] HybridClock initialized: now 1790677712892818 us; error 46 us; skew 500 ppm
Sep 29 10:28:32 dist-test-slave-9cj5 krb5kdc[26904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677712, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:32.894788 27522 init.cc:377] Logged in from keytab as oryx/127.20.92.60@KRBTEST.COM (short username oryx)
I20260929 10:28:32.895478 27522 webserver.cc:460] Webserver started at http://127.20.92.60:40097/ using document root <none> and password file <none>
I20260929 10:28:32.895702 27522 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:32.895753 27522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:32.897383 27522 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:28:32.898590 27716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:32.898872 27522 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:28:32.898985 27522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-2/wal
uuid: "47387bc665724e9792490842b4ffc9ae"
format_stamp: "Formatted at 2026-09-29 10:28:32 on dist-test-slave-9cj5"
server_key: "147b8a448b0df85badecede698d7f649"
server_key_iv: "6f8b41ac6de212345a64602224cc37da"
server_key_version: "encryptionkey@0"
I20260929 10:28:32.899115 27522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260929 10:28:32.929703 27695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:32.930332 27695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:32.930501 27695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 10:28:32.934062 27522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
W20260929 10:28:32.935623 27695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:32.935709 27695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:32.935833 27695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:28:32.936743 27522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:32.942028 27522 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.60:46243
I20260929 10:28:32.942061 27771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.60:46243 every 8 connection(s)
I20260929 10:28:32.942659 27522 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/master-2/data/info.pb
I20260929 10:28:32.943852 27695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:45279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.1.keytab
--principal=oryx/127.20.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:35207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=38309
--tserver_master_addrs=127.20.92.62:38271,127.20.92.61:41373,127.20.92.60:46243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:32.944555 27772 sys_catalog.cc:263] Verifying existing consensus state
I20260929 10:28:32.945339 27695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:32.945680 27772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae: Bootstrap starting.
I20260929 10:28:32.946738 27695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:28:32.957526 27772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:28:32.957890 27772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae: Bootstrap complete.
I20260929 10:28:32.958552 27772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: NON_VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: true } }
I20260929 10:28:32.958731 27772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 47387bc665724e9792490842b4ffc9ae, State: Initialized, Role: LEARNER
I20260929 10:28:32.958966 27772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: NON_VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: true } }
W20260929 10:28:32.959553 27776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:32.959573 27777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:32.959549 27779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:32.960052 27772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:28:32.960371 27781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: NON_VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: true } } }
I20260929 10:28:32.960494 27781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [sys.catalog]: This master's current role is: LEARNER
I20260929 10:28:32.960884 27695 server_base.cc:1034] running on GCE node
I20260929 10:28:32.961511 27695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:32.962416 27695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:32.963511 27792 catalog_manager.cc:1259] Loaded cluster ID: e20e448549d846c1b8d5f05daf6c93da
I20260929 10:28:32.963567 27792 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae: loading cluster ID for follower catalog manager: success
I20260929 10:28:32.965706 27792 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae: acquiring CA information for follower catalog manager: success
I20260929 10:28:32.967032 27792 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 10:28:32.969910 27695 hybrid_clock.cc:648] HybridClock initialized: now 1790677712969865 us; error 49 us; skew 500 ppm
Sep 29 10:28:32 dist-test-slave-9cj5 krb5kdc[26904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677712, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:32.973013 27695 init.cc:377] Logged in from keytab as oryx/127.20.92.1@KRBTEST.COM (short username oryx)
I20260929 10:28:32.974027 27695 webserver.cc:460] Webserver started at http://127.20.92.1:38309/ using document root <none> and password file <none>
I20260929 10:28:32.974689 27695 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:32.974810 27695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:32.978847 27695 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260929 10:28:32.980953 27798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:32.982307 27695 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:28:32.982452 27695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "b2c6034747b64db3910457be844cc69b"
format_stamp: "Formatted at 2026-09-29 10:28:31 on dist-test-slave-9cj5"
server_key: "e5a2ae9597c71c5e0c415aa6a7754b60"
server_key_iv: "1d2806ded52e1a40128d87f254d88f52"
server_key_version: "encryptionkey@0"
I20260929 10:28:32.983049 27695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:33.008951 27695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:33.013126 27695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:33.013346 27695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:33.014031 27695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:33.014910 27695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:33.014978 27695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:33.015046 27695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:33.015092 27695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:33.026255 27695 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:35207
I20260929 10:28:33.026301 27911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:35207 every 8 connection(s)
I20260929 10:28:33.027298 27695 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
I20260929 10:28:33.035928 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 27695
I20260929 10:28:33.036140 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 27145
Sep 29 10:28:33 dist-test-slave-9cj5 krb5kdc[26904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677712, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM
Sep 29 10:28:33 dist-test-slave-9cj5 krb5kdc[26904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677712, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.60@KRBTEST.COM
Sep 29 10:28:33 dist-test-slave-9cj5 krb5kdc[26904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677712, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.61@KRBTEST.COM
I20260929 10:28:33.043917 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:39893
--local_ip_for_outbound_sockets=127.20.92.2
--tserver_master_addrs=127.20.92.62:38271
--webserver_port=41087
--webserver_interface=127.20.92.2
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.2.keytab
--principal=oryx/127.20.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:45279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.92.62:38271,127.20.92.61:41373
--tserver_master_addrs=127.20.92.62:38271,127.20.92.61:41373,127.20.92.60:46243 with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:28:33.046080 26988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.029256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:37947 (local address 127.20.92.62:38271)
0929 10:28:33.029452 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:33.029460 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:33.030475 (+ 1015us) server_negotiation.cc:407] Connection header received
0929 10:28:33.031637 (+ 1162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:33.031641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:33.031702 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:33.031796 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:33.033565 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.034414 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.036190 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.036573 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.036656 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.040958 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:33.040965 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:33.040971 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:33.041011 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:33.043084 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:33.043792 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:33.043799 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:33.043804 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:33.043896 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:33.044622 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:33.044628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:33.044632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:33.044914 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:33.045007 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:33.045716 (+ 709us) server_negotiation.cc:299] Negotiation successful
0929 10:28:33.045899 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 10:28:33.046079 27673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.029394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:43339 (local address 127.20.92.61:41373)
0929 10:28:33.029514 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:33.029518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:33.030479 (+ 961us) server_negotiation.cc:407] Connection header received
0929 10:28:33.031789 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:33.031792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:33.031845 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:33.031952 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:33.033543 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.034224 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.036190 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.036572 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.036657 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.042244 (+ 5587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:33.042250 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:33.042254 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:33.042292 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:33.043968 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:33.044474 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:33.044478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:33.044480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:33.044565 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:33.044866 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:33.044870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:33.044872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:33.045057 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:33.045163 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:33.045702 (+ 539us) server_negotiation.cc:299] Negotiation successful
0929 10:28:33.045872 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 10:28:33.047328 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.029817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:41373 (local address 127.20.92.1:43339)
0929 10:28:33.030315 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:33.030370 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:33.031401 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:33.032090 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:33.032096 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:33.032762 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:33.033382 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:33.033398 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.034376 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.034379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:33.036035 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:33.036047 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.036842 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.036845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:33.037136 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.038365 (+ 1229us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:33.038383 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:33.042020 (+ 3637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:33.044127 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:33.044132 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:33.044134 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:33.044382 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:33.044671 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:33.044674 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:33.044676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:33.044765 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:33.045165 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:33.045168 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:33.045434 (+ 266us) client_negotiation.cc:769] Sending connection context
0929 10:28:33.045666 (+ 232us) client_negotiation.cc:240] Negotiation successful
0929 10:28:33.046388 (+ 722us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":486,"spinlock_wait_cycles":18816,"thread_start_us":140,"threads_started":1}
I20260929 10:28:33.047524 27916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.029817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.20.92.1:37947)
0929 10:28:33.030314 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:33.030369 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:33.031401 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:33.032087 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:33.032094 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:33.033103 (+ 1009us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:33.033393 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:33.033402 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.034525 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.034528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:33.036042 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:33.036047 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.036837 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.036841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:33.037132 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.038495 (+ 1363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:33.038514 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:33.040604 (+ 2090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:33.043230 (+ 2626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:33.043241 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:33.043263 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:33.043650 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:33.044282 (+ 632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:33.044288 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:33.044293 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:33.044507 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:33.045095 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:33.045105 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:33.045439 (+ 334us) client_negotiation.cc:769] Sending connection context
0929 10:28:33.047073 (+ 1634us) client_negotiation.cc:240] Negotiation successful
0929 10:28:33.047238 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":338,"spinlock_wait_cycles":16128,"thread_start_us":146,"threads_started":1}
I20260929 10:28:33.048790 27914 heartbeater.cc:347] Connected to a master server at 127.20.92.61:41373
I20260929 10:28:33.048797 27913 heartbeater.cc:347] Connected to a master server at 127.20.92.62:38271
I20260929 10:28:33.049064 27913 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:33.049661 27913 heartbeater.cc:511] Master 127.20.92.62:38271 requested a full tablet report, sending...
I20260929 10:28:33.050031 27918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.030898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:46243 (local address 127.20.92.1:34691)
0929 10:28:33.031745 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:33.031760 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:33.031846 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:33.032855 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:33.032858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:33.033109 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:33.033394 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:33.033402 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.034900 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.034906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:33.035917 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:33.035929 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.036793 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.036800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:33.037146 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.038150 (+ 1004us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:33.038184 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:33.041591 (+ 3407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:33.047547 (+ 5956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:33.047551 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:33.047553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:33.048623 (+ 1070us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:33.048965 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:33.048968 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:33.048970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:33.049051 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:33.049527 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:33.049530 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:33.049613 (+ 83us) client_negotiation.cc:769] Sending connection context
0929 10:28:33.049714 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 10:28:33.049871 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"mutex_wait_us":166,"spinlock_wait_cycles":5376,"thread_start_us":552,"threads_started":1}
I20260929 10:28:33.049064 27914 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:33.050362 27914 heartbeater.cc:511] Master 127.20.92.61:41373 requested a full tablet report, sending...
I20260929 10:28:33.050536 27919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.031467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:34691 (local address 127.20.92.60:46243)
0929 10:28:33.032412 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:33.032420 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:33.032440 (+ 20us) server_negotiation.cc:407] Connection header received
0929 10:28:33.032492 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:33.032497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:33.032551 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:33.032661 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:33.033562 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.034793 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.036067 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.036506 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.036610 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.045467 (+ 8857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:33.045472 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:33.045474 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:33.045504 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:33.047413 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:33.048747 (+ 1334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:33.048751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:33.048753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:33.048854 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:33.049154 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:33.049159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:33.049160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:33.049392 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:33.049485 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:33.050221 (+ 736us) server_negotiation.cc:299] Negotiation successful
0929 10:28:33.050371 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":77,"threads_started":1}
I20260929 10:28:33.051556 26945 ts_manager.cc:194] Re-registered known tserver with Master: b2c6034747b64db3910457be844cc69b (127.20.92.1:35207)
I20260929 10:28:33.051596 27551 ts_manager.cc:194] Registered new tserver with Master: b2c6034747b64db3910457be844cc69b (127.20.92.1:35207)
I20260929 10:28:33.052449 26945 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.92.1@KRBTEST.COM'} at 127.20.92.1:37947
I20260929 10:28:33.052731 27912 heartbeater.cc:347] Connected to a master server at 127.20.92.60:46243
I20260929 10:28:33.052815 27912 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:33.053005 27912 heartbeater.cc:511] Master 127.20.92.60:46243 requested a full tablet report, sending...
I20260929 10:28:33.054304 27736 ts_manager.cc:194] Registered new tserver with Master: b2c6034747b64db3910457be844cc69b (127.20.92.1:35207)
I20260929 10:28:33.098054 27408 heartbeater.cc:503] Master 127.20.92.62:38271 was elected leader, sending a full tablet report...
W20260929 10:28:33.169108 27920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:33.169409 27920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:33.169456 27920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:33.173836 27920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:33.173905 27920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:33.173970 27920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:28:33.179694 27920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:45279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.2.keytab
--principal=oryx/127.20.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:39893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=41087
--tserver_master_addrs=127.20.92.62:38271,127.20.92.61:41373,127.20.92.60:46243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:33.180903 27920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:33.182042 27920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:33.192543 27926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:33.192541 27928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:33.192541 27925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:33.192782 27920 server_base.cc:1034] running on GCE node
I20260929 10:28:33.193338 27920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:33.194115 27920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:33.195339 27920 hybrid_clock.cc:648] HybridClock initialized: now 1790677713195305 us; error 47 us; skew 500 ppm
Sep 29 10:28:33 dist-test-slave-9cj5 krb5kdc[26904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677713, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:33.198714 27920 init.cc:377] Logged in from keytab as oryx/127.20.92.2@KRBTEST.COM (short username oryx)
I20260929 10:28:33.199872 27920 webserver.cc:460] Webserver started at http://127.20.92.2:41087/ using document root <none> and password file <none>
I20260929 10:28:33.200534 27920 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:33.200614 27920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:33.204981 27920 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:28:33.207640 27935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:33.209293 27920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:28:33.209409 27920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "52a6fe68df7942e4bd2675a0075bac88"
format_stamp: "Formatted at 2026-09-29 10:28:31 on dist-test-slave-9cj5"
server_key: "da1cfc704082c4dc8796191daa402f9f"
server_key_iv: "93b8129d5624020378598042d1bdce60"
server_key_version: "encryptionkey@0"
I20260929 10:28:33.210033 27920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:33.221884 27920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:33.226140 27920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:33.226333 27920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:33.226914 27920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:33.227875 27920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:33.227926 27920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:33.227967 27920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:33.227983 27920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:33.239666 27920 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:39893
I20260929 10:28:33.239777 28048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:39893 every 8 connection(s)
I20260929 10:28:33.240882 27920 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
I20260929 10:28:33.245134 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 27920
I20260929 10:28:33.245335 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 27279
I20260929 10:28:33.254626 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:40193
--local_ip_for_outbound_sockets=127.20.92.3
--tserver_master_addrs=127.20.92.62:38271
--webserver_port=41593
--webserver_interface=127.20.92.3
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.3.keytab
--principal=oryx/127.20.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:45279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.92.62:38271,127.20.92.61:41373
--tserver_master_addrs=127.20.92.62:38271,127.20.92.61:41373,127.20.92.60:46243 with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 10:28:33 dist-test-slave-9cj5 krb5kdc[26904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677713, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.60@KRBTEST.COM
Sep 29 10:28:33 dist-test-slave-9cj5 krb5kdc[26904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677713, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM
Sep 29 10:28:33 dist-test-slave-9cj5 krb5kdc[26904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677713, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.61@KRBTEST.COM
I20260929 10:28:33.262220 27673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.243099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:34941 (local address 127.20.92.61:41373)
0929 10:28:33.243361 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:33.243366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:33.245779 (+ 2413us) server_negotiation.cc:407] Connection header received
0929 10:28:33.246459 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:33.246466 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:33.246539 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:33.246618 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:33.249143 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.250022 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.252156 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.252506 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.252592 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.258025 (+ 5433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:33.258029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:33.258033 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:33.258070 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:33.259952 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:33.260519 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:33.260523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:33.260524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:33.260615 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:33.260924 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:33.260927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:33.260929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:33.261181 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:33.261247 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:33.261924 (+ 677us) server_negotiation.cc:299] Negotiation successful
0929 10:28:33.262054 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 10:28:33.262344 27919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.245463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:45439 (local address 127.20.92.60:46243)
0929 10:28:33.245589 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:33.245594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:33.245611 (+ 17us) server_negotiation.cc:407] Connection header received
0929 10:28:33.246502 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:33.246506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:33.246570 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:33.246630 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:33.249371 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.250192 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.252156 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.252528 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.252598 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.256710 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:33.256715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:33.256718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:33.256754 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:33.259504 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:33.260247 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:33.260254 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:33.260258 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:33.260371 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:33.260756 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:33.260760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:33.260762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:33.261052 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:33.261146 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:33.261879 (+ 733us) server_negotiation.cc:299] Negotiation successful
0929 10:28:33.262070 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 10:28:33.263123 26988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.243101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:44537 (local address 127.20.92.62:38271)
0929 10:28:33.243385 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:33.243392 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:33.244724 (+ 1332us) server_negotiation.cc:407] Connection header received
0929 10:28:33.246502 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:33.246506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:33.246570 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:33.246630 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:33.249886 (+ 3256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.250929 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.252507 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.253010 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.253120 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.256952 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:33.256959 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:33.256965 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:33.257009 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:33.259310 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:33.260095 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:33.260099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:33.260100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:33.260196 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:33.260672 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:33.260678 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:33.260683 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:33.260948 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:33.261043 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:33.262832 (+ 1789us) server_negotiation.cc:299] Negotiation successful
0929 10:28:33.262962 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 10:28:33.263854 28053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.243744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.20.92.2:44537)
0929 10:28:33.244531 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:33.244603 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:33.246059 (+ 1456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:33.247129 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:33.247141 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:33.248046 (+ 905us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:33.248938 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:33.248969 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.251095 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.251102 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:33.252334 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:33.252349 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.253337 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.253341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:33.254107 (+ 766us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.254683 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:33.254702 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:33.256737 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:33.259475 (+ 2738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:33.259482 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:33.259505 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:33.259972 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:33.260321 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:33.260324 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:33.260327 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:33.260544 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:33.261098 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:33.261105 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:33.261487 (+ 382us) client_negotiation.cc:769] Sending connection context
0929 10:28:33.261846 (+ 359us) client_negotiation.cc:240] Negotiation successful
0929 10:28:33.262232 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":518,"spinlock_wait_cycles":5248,"thread_start_us":191,"threads_started":1}
I20260929 10:28:33.264187 28054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.245052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:46243 (local address 127.20.92.2:45439)
0929 10:28:33.245362 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:33.245411 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:33.246059 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:33.247129 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:33.247141 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:33.248171 (+ 1030us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:33.249248 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:33.249260 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.250334 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.250337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:33.252000 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:33.252012 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.252836 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.252839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:33.253126 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.254206 (+ 1080us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:33.254244 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:33.256430 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:33.259667 (+ 3237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:33.259673 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:33.259677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:33.260081 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:33.260521 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:33.260524 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:33.260526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:33.260631 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:33.261255 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:33.261260 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:33.261491 (+ 231us) client_negotiation.cc:769] Sending connection context
0929 10:28:33.262571 (+ 1080us) client_negotiation.cc:240] Negotiation successful
0929 10:28:33.262767 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":787,"spinlock_wait_cycles":54272,"thread_start_us":99,"threads_started":1}
I20260929 10:28:33.263841 28055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.243744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:41373 (local address 127.20.92.2:34941)
0929 10:28:33.245639 (+ 1895us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:33.245664 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:33.246088 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:33.247130 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:33.247143 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:33.248036 (+ 893us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:33.248947 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:33.248967 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.250267 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.250272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:33.252000 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:33.252012 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.252836 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.252839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:33.253130 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.256138 (+ 3008us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:33.256158 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:33.257829 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:33.260104 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:33.260108 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:33.260110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:33.260366 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:33.260729 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:33.260734 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:33.260735 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:33.260820 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:33.261344 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:33.261347 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:33.261487 (+ 140us) client_negotiation.cc:769] Sending connection context
0929 10:28:33.262473 (+ 986us) client_negotiation.cc:240] Negotiation successful
0929 10:28:33.262633 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1507,"mutex_wait_us":258,"spinlock_wait_cycles":76416,"thread_start_us":168,"threads_started":1}
I20260929 10:28:33.265377 28049 heartbeater.cc:347] Connected to a master server at 127.20.92.60:46243
I20260929 10:28:33.265523 28050 heartbeater.cc:347] Connected to a master server at 127.20.92.62:38271
I20260929 10:28:33.265694 28049 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:33.265374 28051 heartbeater.cc:347] Connected to a master server at 127.20.92.61:41373
I20260929 10:28:33.266001 28051 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:33.265693 28050 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:33.266467 28050 heartbeater.cc:511] Master 127.20.92.62:38271 requested a full tablet report, sending...
I20260929 10:28:33.266546 28049 heartbeater.cc:511] Master 127.20.92.60:46243 requested a full tablet report, sending...
I20260929 10:28:33.266477 28051 heartbeater.cc:511] Master 127.20.92.61:41373 requested a full tablet report, sending...
I20260929 10:28:33.268090 27551 ts_manager.cc:194] Registered new tserver with Master: 52a6fe68df7942e4bd2675a0075bac88 (127.20.92.2:39893)
I20260929 10:28:33.269130 27736 ts_manager.cc:194] Registered new tserver with Master: 52a6fe68df7942e4bd2675a0075bac88 (127.20.92.2:39893)
I20260929 10:28:33.269830 26945 ts_manager.cc:194] Re-registered known tserver with Master: 52a6fe68df7942e4bd2675a0075bac88 (127.20.92.2:39893)
I20260929 10:28:33.270717 26945 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.92.2@KRBTEST.COM'} at 127.20.92.2:44537
I20260929 10:28:33.271740 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.267646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:46243 (local address 127.0.0.1:47140)
0929 10:28:33.267900 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:33.267918 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:33.268020 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:33.268530 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:33.268534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:33.268556 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:33.268656 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:33.268661 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.269619 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.269623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:33.270530 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:33.270541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.271339 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.271343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:33.271431 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.271453 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 10:28:33.271534 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 10:28:33.271613 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260929 10:28:33.271739 27919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.267691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47140 (local address 127.20.92.60:46243)
0929 10:28:33.268221 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:33.268243 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:33.268261 (+ 18us) server_negotiation.cc:407] Connection header received
0929 10:28:33.268318 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:33.268323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:33.268399 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:33.268518 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:33.268782 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.269487 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.270673 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.271201 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.271331 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.271481 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:33.271549 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 10:28:33.271619 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425}
I20260929 10:28:33.272358 27746 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 10:28:33.272755 27781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47d8539835204349b4a1620d4eabe0a7. Latest consensus state: current_term: 1 leader_uuid: "47d8539835204349b4a1620d4eabe0a7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: NON_VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: true } } }
I20260929 10:28:33.272913 27781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [sys.catalog]: This master's current role is: LEARNER
W20260929 10:28:33.378486 28056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:33.378777 28056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:33.378821 28056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:33.383123 28056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:33.383194 28056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:33.383294 28056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:28:33.388994 28056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:45279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.3.keytab
--principal=oryx/127.20.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:40193
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=41593
--tserver_master_addrs=127.20.92.62:38271,127.20.92.61:41373,127.20.92.60:46243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:33.390425 28056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:33.391629 28056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:33.402697 28062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:33.402697 28061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:33.402872 28064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:33.403159 28056 server_base.cc:1034] running on GCE node
I20260929 10:28:33.403666 28056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:33.404349 28056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:33.405534 28056 hybrid_clock.cc:648] HybridClock initialized: now 1790677713405484 us; error 61 us; skew 500 ppm
Sep 29 10:28:33 dist-test-slave-9cj5 krb5kdc[26904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677713, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:33.409380 28056 init.cc:377] Logged in from keytab as oryx/127.20.92.3@KRBTEST.COM (short username oryx)
I20260929 10:28:33.410499 28056 webserver.cc:460] Webserver started at http://127.20.92.3:41593/ using document root <none> and password file <none>
I20260929 10:28:33.411101 28056 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:33.411180 28056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:33.416342 28056 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:28:33.418956 28071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:33.420732 28056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:28:33.420902 28056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "77ff8f0666ff4666b205c8addfa5706f"
format_stamp: "Formatted at 2026-09-29 10:28:32 on dist-test-slave-9cj5"
server_key: "3200372c9234be73fff417316d5e73f8"
server_key_iv: "e96b27a23118490744e77699afed46dd"
server_key_version: "encryptionkey@0"
I20260929 10:28:33.421562 28056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:33.444480 28056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:33.448856 28056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:33.449062 28056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:33.449703 28056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:33.450659 28056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:33.450731 28056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:33.450794 28056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:33.450824 28056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:33.459646 28056 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:40193
I20260929 10:28:33.459708 28184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:40193 every 8 connection(s)
I20260929 10:28:33.460865 28056 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:28:33.463202 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 28056
Sep 29 10:28:33 dist-test-slave-9cj5 krb5kdc[26904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677713, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.60@KRBTEST.COM
Sep 29 10:28:33 dist-test-slave-9cj5 krb5kdc[26904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677713, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.61@KRBTEST.COM
I20260929 10:28:33.476154 28197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.465195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49186 (local address 127.20.92.61:41373)
0929 10:28:33.465417 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:33.465421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:33.465463 (+ 42us) server_negotiation.cc:407] Connection header received
0929 10:28:33.465643 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:33.465648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:33.465712 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:33.465837 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:33.466914 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.467720 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.468989 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.469344 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.469415 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.471318 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:33.471322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:33.471325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:33.471353 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:33.473826 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:33.474337 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:33.474341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:33.474342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:33.474428 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:33.474729 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:33.474733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:33.474734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:33.475024 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:33.475149 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:33.475759 (+ 610us) server_negotiation.cc:299] Negotiation successful
0929 10:28:33.475950 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":164,"thread_start_us":76,"threads_started":1}
Sep 29 10:28:33 dist-test-slave-9cj5 krb5kdc[26904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677713, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM
I20260929 10:28:33.478899 28199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.465618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36376 (local address 127.20.92.62:38271)
0929 10:28:33.465902 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:33.465907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:33.467299 (+ 1392us) server_negotiation.cc:407] Connection header received
0929 10:28:33.467496 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:33.467499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:33.467545 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:33.467604 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:33.468544 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.469450 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.470673 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.471010 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.471079 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.473319 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:33.473326 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:33.473331 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:33.473366 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:33.476462 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:33.477001 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:33.477004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:33.477008 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:33.477110 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:33.477442 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:33.477448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:33.477452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:33.477687 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:33.477914 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:33.478541 (+ 627us) server_negotiation.cc:299] Negotiation successful
0929 10:28:33.478676 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":119,"threads_started":1}
I20260929 10:28:33.479338 27919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.462794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:58871 (local address 127.20.92.60:46243)
0929 10:28:33.463101 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:33.463109 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:33.466478 (+ 3369us) server_negotiation.cc:407] Connection header received
0929 10:28:33.466583 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:33.466587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:33.466639 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:33.466729 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:33.467842 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.468640 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.470641 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.471002 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.471079 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.474441 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:33.474446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:33.474450 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:33.474482 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:33.476710 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:33.477346 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:33.477352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:33.477354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:33.477448 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:33.477944 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:33.477948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:33.477951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:33.478190 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:33.478284 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:33.479033 (+ 749us) server_negotiation.cc:299] Negotiation successful
0929 10:28:33.479157 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207}
I20260929 10:28:33.481189 28189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.463377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:46243 (local address 127.20.92.3:58871)
0929 10:28:33.466368 (+ 2991us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:33.466386 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:33.466501 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:33.466842 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:33.466845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:33.467022 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:33.467573 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:33.467596 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.468843 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.468846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:33.470472 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:33.470483 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.471284 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.471289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:33.471528 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.472103 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:33.472130 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:33.474228 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:33.476836 (+ 2608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:33.476843 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:33.476861 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:33.477224 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:33.477569 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:33.477573 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:33.477577 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:33.477818 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:33.478353 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:33.478363 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:33.478742 (+ 379us) client_negotiation.cc:769] Sending connection context
0929 10:28:33.479731 (+ 989us) client_negotiation.cc:240] Negotiation successful
0929 10:28:33.480087 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2850,"mutex_wait_us":106,"spinlock_wait_cycles":15104,"thread_start_us":147,"threads_started":1}
I20260929 10:28:33.482827 28185 heartbeater.cc:347] Connected to a master server at 127.20.92.60:46243
I20260929 10:28:33.483067 28185 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:33.483623 28185 heartbeater.cc:511] Master 127.20.92.60:46243 requested a full tablet report, sending...
I20260929 10:28:33.484776 27736 ts_manager.cc:194] Registered new tserver with Master: 77ff8f0666ff4666b205c8addfa5706f (127.20.92.3:40193)
I20260929 10:28:33.485055 26988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.464596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:40331 (local address 127.20.92.62:38271)
0929 10:28:33.464718 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:33.464722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:33.467048 (+ 2326us) server_negotiation.cc:407] Connection header received
0929 10:28:33.467225 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:33.467228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:33.467274 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:33.467334 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:33.469264 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.470002 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.474988 (+ 4986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.475326 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.475407 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.480401 (+ 4994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:33.480405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:33.480407 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:33.480432 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:33.483036 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:33.483581 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:33.483584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:33.483586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:33.483669 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:33.483985 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:33.483989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:33.483991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:33.484214 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:33.484317 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:33.484723 (+ 406us) server_negotiation.cc:299] Negotiation successful
0929 10:28:33.484859 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 10:28:33.482489 26945 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36376:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 10:28:33.481400 27673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.462950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:44739 (local address 127.20.92.61:41373)
0929 10:28:33.463205 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:33.463209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:33.464764 (+ 1555us) server_negotiation.cc:407] Connection header received
0929 10:28:33.465792 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:33.465796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:33.465843 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:33.465896 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:33.468118 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.468880 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.471072 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.471410 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.471488 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.474967 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:33.474973 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:33.474979 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:33.475029 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:33.478054 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:33.478527 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:33.478530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:33.478531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:33.478611 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:33.478937 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:33.478943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:33.478946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:33.479167 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:33.479249 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:33.479618 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 10:28:33.480987 (+ 1369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"spinlock_wait_cycles":3456}
I20260929 10:28:33.485649 28195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.464452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.20.92.3:40331)
0929 10:28:33.466973 (+ 2521us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:33.466986 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:33.467108 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:33.467496 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:33.467499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:33.467672 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:33.467870 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:33.467878 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.472018 (+ 4140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.472023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:33.472981 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:33.472993 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.475582 (+ 2589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.475585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:33.475819 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.476548 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:33.476566 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:33.480238 (+ 3672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:33.483184 (+ 2946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:33.483187 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:33.483191 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:33.483455 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:33.483781 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:33.483784 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:33.483786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:33.483883 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:33.484384 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:33.484390 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:33.484517 (+ 127us) client_negotiation.cc:769] Sending connection context
0929 10:28:33.485249 (+ 732us) client_negotiation.cc:240] Negotiation successful
0929 10:28:33.485429 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2462,"mutex_wait_us":58,"thread_start_us":103,"threads_started":1}
I20260929 10:28:33.486399 28186 heartbeater.cc:347] Connected to a master server at 127.20.92.62:38271
I20260929 10:28:33.486490 28186 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:33.486665 28186 heartbeater.cc:511] Master 127.20.92.62:38271 requested a full tablet report, sending...
I20260929 10:28:33.486737 28190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.463377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:41373 (local address 127.20.92.3:44739)
0929 10:28:33.464509 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:33.464571 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:33.465527 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:33.466330 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:33.466346 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:33.466831 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:33.468009 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:33.468015 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.469010 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.469013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:33.470921 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:33.470935 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.471641 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.471647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:33.471779 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.473162 (+ 1383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:33.473181 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:33.474840 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:33.478163 (+ 3323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:33.478166 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:33.478168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:33.478421 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:33.478719 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:33.478722 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:33.478724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:33.478802 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:33.479324 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:33.479326 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:33.479428 (+ 102us) client_negotiation.cc:769] Sending connection context
0929 10:28:33.486405 (+ 6977us) client_negotiation.cc:240] Negotiation successful
0929 10:28:33.486556 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"mutex_wait_us":1699,"thread_start_us":117,"threads_started":1}
W20260929 10:28:33.487159 26945 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:28:33.487365 28187 heartbeater.cc:347] Connected to a master server at 127.20.92.61:41373
I20260929 10:28:33.487478 28187 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:33.487644 28187 heartbeater.cc:511] Master 127.20.92.61:41373 requested a full tablet report, sending...
I20260929 10:28:33.488080 26949 ts_manager.cc:194] Re-registered known tserver with Master: 77ff8f0666ff4666b205c8addfa5706f (127.20.92.3:40193)
I20260929 10:28:33.488452 28200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.466007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47156 (local address 127.20.92.60:46243)
0929 10:28:33.466256 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:33.466260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:33.471965 (+ 5705us) server_negotiation.cc:407] Connection header received
0929 10:28:33.472008 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:33.472022 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:33.472076 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:33.472164 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:33.478830 (+ 6666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.479831 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.481003 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.481669 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.481809 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.483946 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:33.483952 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:33.483958 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:33.483986 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:33.486065 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:33.486634 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:33.486639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:33.486641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:33.486724 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:33.487077 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:33.487082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:33.487084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:33.487317 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:33.487468 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:33.488150 (+ 682us) server_negotiation.cc:299] Negotiation successful
0929 10:28:33.488302 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":79,"threads_started":1}
I20260929 10:28:33.488744 26949 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.92.3@KRBTEST.COM'} at 127.20.92.3:40331
I20260929 10:28:33.489503 27551 ts_manager.cc:194] Registered new tserver with Master: 77ff8f0666ff4666b205c8addfa5706f (127.20.92.3:40193)
I20260929 10:28:33.490485 27512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47d8539835204349b4a1620d4eabe0a7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: NON_VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: true } } }
I20260929 10:28:33.490610 27512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:33.496533 28204 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7: attempting to promote NON_VOTER 47387bc665724e9792490842b4ffc9ae to VOTER
I20260929 10:28:33.496963 28204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: false } }
I20260929 10:28:33.498698 27561 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [term 1 FOLLOWER]: Refusing update from remote peer 47d8539835204349b4a1620d4eabe0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 10:28:33.498744 27745 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [term 1 LEARNER]: Refusing update from remote peer 47d8539835204349b4a1620d4eabe0a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 10:28:33.499763 28204 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 10:28:33.500218 28204 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260929 10:28:33.502388 27512 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 47387bc665724e9792490842b4ffc9ae (127.20.92.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: false } } }
I20260929 10:28:33.502782 27561 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 47387bc665724e9792490842b4ffc9ae (127.20.92.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: false } } }
I20260929 10:28:33.503978 27512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "47d8539835204349b4a1620d4eabe0a7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: false } } }
I20260929 10:28:33.504117 27512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:33.504273 27512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47d8539835204349b4a1620d4eabe0a7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: false } } }
I20260929 10:28:33.504383 27512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:33.505235 28212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47d8539835204349b4a1620d4eabe0a7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: false } } }
I20260929 10:28:33.505498 28212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:28:33.506692 28203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47d8539835204349b4a1620d4eabe0a7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: false } } }
I20260929 10:28:33.506827 28203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:33.508616 27746 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 47387bc665724e9792490842b4ffc9ae (127.20.92.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: false } } }
I20260929 10:28:33.511564 27781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47d8539835204349b4a1620d4eabe0a7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: false } } }
I20260929 10:28:33.511693 27781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:28:33.515230 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.507583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:39893 (local address 127.0.0.1:55160)
0929 10:28:33.508087 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:33.508113 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:33.508211 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:33.510517 (+ 2306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:33.510524 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:33.510549 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:33.510675 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:33.510683 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.512040 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.512047 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:33.513572 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:33.513587 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.514775 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.514779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:33.514895 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.514924 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 10:28:33.515008 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 10:28:33.515087 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409}
I20260929 10:28:33.516029 28213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.508413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55160 (local address 127.20.92.2:39893)
0929 10:28:33.508775 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:33.508792 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:33.508819 (+ 27us) server_negotiation.cc:407] Connection header received
0929 10:28:33.508902 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:33.508908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:33.509145 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:33.509328 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:33.510795 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.511941 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.513726 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.514638 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.514759 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.514906 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:33.515747 (+ 841us) server_negotiation.cc:299] Negotiation successful
0929 10:28:33.515875 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":165,"threads_started":1}
I20260929 10:28:33.519301 28215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.508951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:40193 (local address 127.0.0.1:52592)
0929 10:28:33.511001 (+ 2050us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:33.511016 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:33.511203 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:33.511935 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:33.511937 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:33.511955 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:33.513837 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:33.513844 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.515213 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.515217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:33.517338 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:33.517352 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.518855 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.518858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:33.518953 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.518976 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 10:28:33.519063 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 10:28:33.519138 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1984,"thread_start_us":94,"threads_started":1}
I20260929 10:28:33.519402 28216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.509858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52592 (local address 127.20.92.3:40193)
0929 10:28:33.510154 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:33.510166 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:33.511105 (+ 939us) server_negotiation.cc:407] Connection header received
0929 10:28:33.511349 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:33.511355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:33.511574 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:33.511759 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:33.513974 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.515057 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.517597 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.518513 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.518599 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.518750 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:33.519131 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 10:28:33.519243 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":114,"threads_started":1}
I20260929 10:28:33.519459 27983 tablet_service.cc:1467] Processing CreateTablet for tablet dd97944ac00e424e8b1e2a8092a3c71f (DEFAULT_TABLE table=test-table [id=8cbf4c7aafac4f6aa9733b4daf19c4e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:33.522852 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.509261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:35207 (local address 127.0.0.1:49626)
0929 10:28:33.515413 (+ 6152us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:33.515430 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:33.515518 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:33.517947 (+ 2429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:33.517951 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:33.517971 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:33.518062 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:33.518068 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.519451 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.519457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:33.520690 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:33.520706 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.522192 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.522196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:33.522463 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.522497 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 10:28:33.522602 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 10:28:33.522687 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6091,"thread_start_us":83,"threads_started":1}
I20260929 10:28:33.522974 28217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.510242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49626 (local address 127.20.92.1:35207)
0929 10:28:33.510488 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:33.510503 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:33.515513 (+ 5010us) server_negotiation.cc:407] Connection header received
0929 10:28:33.515632 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:33.515637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:33.515874 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:33.516060 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:33.518201 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.519298 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.520877 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.522004 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.522168 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.522580 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:33.522648 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 10:28:33.522760 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":86,"threads_started":1}
I20260929 10:28:33.523187 28119 tablet_service.cc:1467] Processing CreateTablet for tablet dd97944ac00e424e8b1e2a8092a3c71f (DEFAULT_TABLE table=test-table [id=8cbf4c7aafac4f6aa9733b4daf19c4e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:33.524427 28119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd97944ac00e424e8b1e2a8092a3c71f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:33.526252 27846 tablet_service.cc:1467] Processing CreateTablet for tablet dd97944ac00e424e8b1e2a8092a3c71f (DEFAULT_TABLE table=test-table [id=8cbf4c7aafac4f6aa9733b4daf19c4e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:33.527479 27846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd97944ac00e424e8b1e2a8092a3c71f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:33.529273 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd97944ac00e424e8b1e2a8092a3c71f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:33.536389 28218 tablet_bootstrap.cc:492] T dd97944ac00e424e8b1e2a8092a3c71f P b2c6034747b64db3910457be844cc69b: Bootstrap starting.
I20260929 10:28:33.538136 28219 tablet_bootstrap.cc:492] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88: Bootstrap starting.
I20260929 10:28:33.538998 28218 tablet_bootstrap.cc:654] T dd97944ac00e424e8b1e2a8092a3c71f P b2c6034747b64db3910457be844cc69b: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:33.540016 28218 log.cc:824] T dd97944ac00e424e8b1e2a8092a3c71f P b2c6034747b64db3910457be844cc69b: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:33.540863 28220 tablet_bootstrap.cc:492] T dd97944ac00e424e8b1e2a8092a3c71f P 77ff8f0666ff4666b205c8addfa5706f: Bootstrap starting.
I20260929 10:28:33.541054 28219 tablet_bootstrap.cc:654] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:33.541908 28219 log.cc:824] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:33.542554 28218 tablet_bootstrap.cc:492] T dd97944ac00e424e8b1e2a8092a3c71f P b2c6034747b64db3910457be844cc69b: No bootstrap required, opened a new log
I20260929 10:28:33.542781 28218 ts_tablet_manager.cc:1397] T dd97944ac00e424e8b1e2a8092a3c71f P b2c6034747b64db3910457be844cc69b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 10:28:33.543462 28220 tablet_bootstrap.cc:654] T dd97944ac00e424e8b1e2a8092a3c71f P 77ff8f0666ff4666b205c8addfa5706f: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:33.544301 28220 log.cc:824] T dd97944ac00e424e8b1e2a8092a3c71f P 77ff8f0666ff4666b205c8addfa5706f: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:33.546595 28220 tablet_bootstrap.cc:492] T dd97944ac00e424e8b1e2a8092a3c71f P 77ff8f0666ff4666b205c8addfa5706f: No bootstrap required, opened a new log
I20260929 10:28:33.546802 28220 ts_tablet_manager.cc:1397] T dd97944ac00e424e8b1e2a8092a3c71f P 77ff8f0666ff4666b205c8addfa5706f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 10:28:33.547583 28219 tablet_bootstrap.cc:492] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88: No bootstrap required, opened a new log
I20260929 10:28:33.547435 28218 raft_consensus.cc:348] T dd97944ac00e424e8b1e2a8092a3c71f P b2c6034747b64db3910457be844cc69b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a6fe68df7942e4bd2675a0075bac88" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39893 } } peers { permanent_uuid: "77ff8f0666ff4666b205c8addfa5706f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40193 } } peers { permanent_uuid: "b2c6034747b64db3910457be844cc69b" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35207 } }
I20260929 10:28:33.547696 28218 raft_consensus.cc:374] T dd97944ac00e424e8b1e2a8092a3c71f P b2c6034747b64db3910457be844cc69b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:33.547744 28218 raft_consensus.cc:729] T dd97944ac00e424e8b1e2a8092a3c71f P b2c6034747b64db3910457be844cc69b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2c6034747b64db3910457be844cc69b, State: Initialized, Role: FOLLOWER
I20260929 10:28:33.547753 28219 ts_tablet_manager.cc:1397] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.003s
I20260929 10:28:33.548382 28218 consensus_queue.cc:260] T dd97944ac00e424e8b1e2a8092a3c71f P b2c6034747b64db3910457be844cc69b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a6fe68df7942e4bd2675a0075bac88" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39893 } } peers { permanent_uuid: "77ff8f0666ff4666b205c8addfa5706f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40193 } } peers { permanent_uuid: "b2c6034747b64db3910457be844cc69b" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35207 } }
I20260929 10:28:33.549331 28218 ts_tablet_manager.cc:1428] T dd97944ac00e424e8b1e2a8092a3c71f P b2c6034747b64db3910457be844cc69b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 10:28:33.550658 27913 heartbeater.cc:503] Master 127.20.92.62:38271 was elected leader, sending a full tablet report...
I20260929 10:28:33.553889 28219 raft_consensus.cc:348] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a6fe68df7942e4bd2675a0075bac88" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39893 } } peers { permanent_uuid: "77ff8f0666ff4666b205c8addfa5706f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40193 } } peers { permanent_uuid: "b2c6034747b64db3910457be844cc69b" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35207 } }
I20260929 10:28:33.554190 28219 raft_consensus.cc:374] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:33.554239 28219 raft_consensus.cc:729] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a6fe68df7942e4bd2675a0075bac88, State: Initialized, Role: FOLLOWER
I20260929 10:28:33.554782 28219 consensus_queue.cc:260] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a6fe68df7942e4bd2675a0075bac88" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39893 } } peers { permanent_uuid: "77ff8f0666ff4666b205c8addfa5706f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40193 } } peers { permanent_uuid: "b2c6034747b64db3910457be844cc69b" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35207 } }
I20260929 10:28:33.555954 28219 ts_tablet_manager.cc:1428] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s
I20260929 10:28:33.556159 28050 heartbeater.cc:503] Master 127.20.92.62:38271 was elected leader, sending a full tablet report...
I20260929 10:28:33.556303 28220 raft_consensus.cc:348] T dd97944ac00e424e8b1e2a8092a3c71f P 77ff8f0666ff4666b205c8addfa5706f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a6fe68df7942e4bd2675a0075bac88" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39893 } } peers { permanent_uuid: "77ff8f0666ff4666b205c8addfa5706f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40193 } } peers { permanent_uuid: "b2c6034747b64db3910457be844cc69b" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35207 } }
I20260929 10:28:33.556567 28220 raft_consensus.cc:374] T dd97944ac00e424e8b1e2a8092a3c71f P 77ff8f0666ff4666b205c8addfa5706f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:33.556618 28220 raft_consensus.cc:729] T dd97944ac00e424e8b1e2a8092a3c71f P 77ff8f0666ff4666b205c8addfa5706f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ff8f0666ff4666b205c8addfa5706f, State: Initialized, Role: FOLLOWER
I20260929 10:28:33.557050 28220 consensus_queue.cc:260] T dd97944ac00e424e8b1e2a8092a3c71f P 77ff8f0666ff4666b205c8addfa5706f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a6fe68df7942e4bd2675a0075bac88" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39893 } } peers { permanent_uuid: "77ff8f0666ff4666b205c8addfa5706f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40193 } } peers { permanent_uuid: "b2c6034747b64db3910457be844cc69b" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35207 } }
I20260929 10:28:33.558109 28220 ts_tablet_manager.cc:1428] T dd97944ac00e424e8b1e2a8092a3c71f P 77ff8f0666ff4666b205c8addfa5706f: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260929 10:28:33.558280 28186 heartbeater.cc:503] Master 127.20.92.62:38271 was elected leader, sending a full tablet report...
W20260929 10:28:33.712334 28188 tablet.cc:2369] T dd97944ac00e424e8b1e2a8092a3c71f P 77ff8f0666ff4666b205c8addfa5706f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:28:33.728396 28225 raft_consensus.cc:482] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:28:33.728740 28225 raft_consensus.cc:504] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a6fe68df7942e4bd2675a0075bac88" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39893 } } peers { permanent_uuid: "77ff8f0666ff4666b205c8addfa5706f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40193 } } peers { permanent_uuid: "b2c6034747b64db3910457be844cc69b" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35207 } }
I20260929 10:28:33.730062 28225 leader_election.cc:288] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77ff8f0666ff4666b205c8addfa5706f (127.20.92.3:40193), b2c6034747b64db3910457be844cc69b (127.20.92.1:35207)
I20260929 10:28:33.734532 28055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.730356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:40193 (local address 127.20.92.2:40605)
0929 10:28:33.730552 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:33.730569 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:33.730711 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:33.731003 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:33.731006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:33.731029 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:33.731171 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:33.731183 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.732175 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.732179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:33.733078 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:33.733088 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.733994 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.733998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:33.734111 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.734184 (+ 73us) client_negotiation.cc:769] Sending connection context
0929 10:28:33.734269 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 10:28:33.734356 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 10:28:33.734548 28216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.730605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:40605 (local address 127.20.92.3:40193)
0929 10:28:33.730746 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:33.730750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:33.730768 (+ 18us) server_negotiation.cc:407] Connection header received
0929 10:28:33.730827 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:33.730831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:33.730884 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:33.730986 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:33.731318 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.732005 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.733249 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.733859 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.733945 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.734001 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:33.734319 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 10:28:33.734397 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 10:28:33.734712 28054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.730690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:35207 (local address 127.20.92.2:46165)
0929 10:28:33.730882 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:33.730904 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:33.730985 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:33.731291 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:33.731296 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:33.731317 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:33.731429 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:33.731435 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.732450 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.732452 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:33.733401 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:33.733411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.734229 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.734233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:33.734363 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.734387 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 10:28:33.734472 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 10:28:33.734536 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
I20260929 10:28:33.734747 28217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.730858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:46165 (local address 127.20.92.1:35207)
0929 10:28:33.731005 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:33.731009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:33.731027 (+ 18us) server_negotiation.cc:407] Connection header received
0929 10:28:33.731084 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:33.731087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:33.731140 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:33.731242 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:33.731696 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.732315 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.733541 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.734101 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.734233 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.734312 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:33.734506 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 10:28:33.734599 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 10:28:33.735419 27866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd97944ac00e424e8b1e2a8092a3c71f" candidate_uuid: "52a6fe68df7942e4bd2675a0075bac88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2c6034747b64db3910457be844cc69b" is_pre_election: true
I20260929 10:28:33.735491 28139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd97944ac00e424e8b1e2a8092a3c71f" candidate_uuid: "52a6fe68df7942e4bd2675a0075bac88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ff8f0666ff4666b205c8addfa5706f" is_pre_election: true
I20260929 10:28:33.735756 27866 raft_consensus.cc:2454] T dd97944ac00e424e8b1e2a8092a3c71f P b2c6034747b64db3910457be844cc69b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52a6fe68df7942e4bd2675a0075bac88 in term 0.
I20260929 10:28:33.735823 28139 raft_consensus.cc:2454] T dd97944ac00e424e8b1e2a8092a3c71f P 77ff8f0666ff4666b205c8addfa5706f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52a6fe68df7942e4bd2675a0075bac88 in term 0.
I20260929 10:28:33.736269 27938 leader_election.cc:302] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52a6fe68df7942e4bd2675a0075bac88, b2c6034747b64db3910457be844cc69b; no voters:
I20260929 10:28:33.736552 28225 raft_consensus.cc:2781] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:28:33.736634 28225 raft_consensus.cc:482] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:28:33.736666 28225 raft_consensus.cc:3037] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:33.737895 28225 raft_consensus.cc:504] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a6fe68df7942e4bd2675a0075bac88" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39893 } } peers { permanent_uuid: "77ff8f0666ff4666b205c8addfa5706f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40193 } } peers { permanent_uuid: "b2c6034747b64db3910457be844cc69b" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35207 } }
I20260929 10:28:33.738276 28225 leader_election.cc:288] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88 [CANDIDATE]: Term 1 election: Requested vote from peers 77ff8f0666ff4666b205c8addfa5706f (127.20.92.3:40193), b2c6034747b64db3910457be844cc69b (127.20.92.1:35207)
I20260929 10:28:33.738786 28139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd97944ac00e424e8b1e2a8092a3c71f" candidate_uuid: "52a6fe68df7942e4bd2675a0075bac88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ff8f0666ff4666b205c8addfa5706f"
I20260929 10:28:33.738791 27866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd97944ac00e424e8b1e2a8092a3c71f" candidate_uuid: "52a6fe68df7942e4bd2675a0075bac88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2c6034747b64db3910457be844cc69b"
I20260929 10:28:33.738905 28139 raft_consensus.cc:3037] T dd97944ac00e424e8b1e2a8092a3c71f P 77ff8f0666ff4666b205c8addfa5706f [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:33.739003 27866 raft_consensus.cc:3037] T dd97944ac00e424e8b1e2a8092a3c71f P b2c6034747b64db3910457be844cc69b [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:33.740069 27866 raft_consensus.cc:2454] T dd97944ac00e424e8b1e2a8092a3c71f P b2c6034747b64db3910457be844cc69b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52a6fe68df7942e4bd2675a0075bac88 in term 1.
I20260929 10:28:33.740125 28139 raft_consensus.cc:2454] T dd97944ac00e424e8b1e2a8092a3c71f P 77ff8f0666ff4666b205c8addfa5706f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52a6fe68df7942e4bd2675a0075bac88 in term 1.
I20260929 10:28:33.740382 27938 leader_election.cc:302] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52a6fe68df7942e4bd2675a0075bac88, b2c6034747b64db3910457be844cc69b; no voters:
I20260929 10:28:33.740568 28225 raft_consensus.cc:2781] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:33.740777 28225 raft_consensus.cc:686] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88 [term 1 LEADER]: Becoming Leader. State: Replica: 52a6fe68df7942e4bd2675a0075bac88, State: Running, Role: LEADER
I20260929 10:28:33.741321 28225 consensus_queue.cc:237] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a6fe68df7942e4bd2675a0075bac88" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39893 } } peers { permanent_uuid: "77ff8f0666ff4666b205c8addfa5706f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40193 } } peers { permanent_uuid: "b2c6034747b64db3910457be844cc69b" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35207 } }
W20260929 10:28:33.742601 28052 tablet.cc:2369] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:28:33.745240 26949 catalog_manager.cc:5495] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52a6fe68df7942e4bd2675a0075bac88 (127.20.92.2). New cstate: current_term: 1 leader_uuid: "52a6fe68df7942e4bd2675a0075bac88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a6fe68df7942e4bd2675a0075bac88" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77ff8f0666ff4666b205c8addfa5706f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2c6034747b64db3910457be844cc69b" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35207 } health_report { overall_health: UNKNOWN } } }
W20260929 10:28:33.779438 27915 tablet.cc:2369] T dd97944ac00e424e8b1e2a8092a3c71f P b2c6034747b64db3910457be844cc69b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:28:33.813536 28213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.809879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55166 (local address 127.20.92.2:39893)
0929 10:28:33.810008 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:33.810018 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:33.810031 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:28:33.810105 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:33.810108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:33.810196 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:33.810293 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:28:33.810545 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.811078 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.811974 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.812237 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.812327 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.812611 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:28:33.813143 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:28:33.813282 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:33.813368 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 10:28:33.813422 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 10:28:33.819800 27866 raft_consensus.cc:1261] T dd97944ac00e424e8b1e2a8092a3c71f P b2c6034747b64db3910457be844cc69b [term 1 FOLLOWER]: Refusing update from remote peer 52a6fe68df7942e4bd2675a0075bac88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:33.819905 28139 raft_consensus.cc:1261] T dd97944ac00e424e8b1e2a8092a3c71f P 77ff8f0666ff4666b205c8addfa5706f [term 1 FOLLOWER]: Refusing update from remote peer 52a6fe68df7942e4bd2675a0075bac88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:33.820600 28225 consensus_queue.cc:1035] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2c6034747b64db3910457be844cc69b" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:28:33.820807 28227 consensus_queue.cc:1035] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff8f0666ff4666b205c8addfa5706f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:28:33.830642 28231 mvcc.cc:205] Tried to move back new op lower bound from 7334615915798503424 to 7334615915487895552. Current Snapshot: MvccSnapshot[applied={T|T < 7334615915798503424}]
I20260929 10:28:33.830716 28232 mvcc.cc:205] Tried to move back new op lower bound from 7334615915798503424 to 7334615915487895552. Current Snapshot: MvccSnapshot[applied={T|T < 7334615915798503424}]
I20260929 10:28:33.840458 26945 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:28:33.840612 26945 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:28:33.845988 26945 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47d8539835204349b4a1620d4eabe0a7: Sending DeleteTablet for 3 replicas of tablet dd97944ac00e424e8b1e2a8092a3c71f
I20260929 10:28:33.846674 27983 tablet_service.cc:1514] Processing DeleteTablet for tablet dd97944ac00e424e8b1e2a8092a3c71f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:33 UTC) from {username='oryx', principal='oryx/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:55160
I20260929 10:28:33.847210 28119 tablet_service.cc:1514] Processing DeleteTablet for tablet dd97944ac00e424e8b1e2a8092a3c71f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:33 UTC) from {username='oryx', principal='oryx/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:52592
I20260929 10:28:33.847330 28240 tablet_replica.cc:330] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88: stopping tablet replica
I20260929 10:28:33.847738 28240 raft_consensus.cc:2229] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:28:33.847741 27846 tablet_service.cc:1514] Processing DeleteTablet for tablet dd97944ac00e424e8b1e2a8092a3c71f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:33 UTC) from {username='oryx', principal='oryx/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:49626
I20260929 10:28:33.848153 28240 raft_consensus.cc:2258] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:33.848541 28241 tablet_replica.cc:330] T dd97944ac00e424e8b1e2a8092a3c71f P 77ff8f0666ff4666b205c8addfa5706f: stopping tablet replica
I20260929 10:28:33.848874 28241 raft_consensus.cc:2229] T dd97944ac00e424e8b1e2a8092a3c71f P 77ff8f0666ff4666b205c8addfa5706f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:28:33.849294 28241 raft_consensus.cc:2258] T dd97944ac00e424e8b1e2a8092a3c71f P 77ff8f0666ff4666b205c8addfa5706f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:33.849890 28240 ts_tablet_manager.cc:1905] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:33.850473 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 26917
I20260929 10:28:33.851346 28241 ts_tablet_manager.cc:1905] T dd97944ac00e424e8b1e2a8092a3c71f P 77ff8f0666ff4666b205c8addfa5706f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:33.853322 28240 ts_tablet_manager.cc:1918] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:28:33.853436 28240 log.cc:1196] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/wal/wals/dd97944ac00e424e8b1e2a8092a3c71f
I20260929 10:28:33.853850 28240 ts_tablet_manager.cc:1939] T dd97944ac00e424e8b1e2a8092a3c71f P 52a6fe68df7942e4bd2675a0075bac88: Deleting consensus metadata
I20260929 10:28:33.854476 28241 ts_tablet_manager.cc:1918] T dd97944ac00e424e8b1e2a8092a3c71f P 77ff8f0666ff4666b205c8addfa5706f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:28:33.854562 28241 log.cc:1196] T dd97944ac00e424e8b1e2a8092a3c71f P 77ff8f0666ff4666b205c8addfa5706f: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/wal/wals/dd97944ac00e424e8b1e2a8092a3c71f
I20260929 10:28:33.854904 28241 ts_tablet_manager.cc:1939] T dd97944ac00e424e8b1e2a8092a3c71f P 77ff8f0666ff4666b205c8addfa5706f: Deleting consensus metadata
I20260929 10:28:33.856624 28242 tablet_replica.cc:330] T dd97944ac00e424e8b1e2a8092a3c71f P b2c6034747b64db3910457be844cc69b: stopping tablet replica
I20260929 10:28:33.857040 28242 raft_consensus.cc:2229] T dd97944ac00e424e8b1e2a8092a3c71f P b2c6034747b64db3910457be844cc69b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:28:33.857462 28242 raft_consensus.cc:2258] T dd97944ac00e424e8b1e2a8092a3c71f P b2c6034747b64db3910457be844cc69b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:33.859373 28242 ts_tablet_manager.cc:1905] T dd97944ac00e424e8b1e2a8092a3c71f P b2c6034747b64db3910457be844cc69b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 10:28:33.859503 28072 connection.cc:535] server connection from 127.0.0.1:52592 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 10:28:33.859897 27936 connection.cc:535] server connection from 127.0.0.1:55160 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 10:28:33.863557 28242 ts_tablet_manager.cc:1918] T dd97944ac00e424e8b1e2a8092a3c71f P b2c6034747b64db3910457be844cc69b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:28:33.863813 28242 log.cc:1196] T dd97944ac00e424e8b1e2a8092a3c71f P b2c6034747b64db3910457be844cc69b: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-0/wal/wals/dd97944ac00e424e8b1e2a8092a3c71f
I20260929 10:28:33.864293 28242 ts_tablet_manager.cc:1939] T dd97944ac00e424e8b1e2a8092a3c71f P b2c6034747b64db3910457be844cc69b: Deleting consensus metadata
W20260929 10:28:33.865768 27799 connection.cc:441] server connection from 127.0.0.1:49626 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49626 (request call id 1) could send its response
I20260929 10:28:33.867089 27673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.848867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49194 (local address 127.20.92.61:41373)
0929 10:28:33.849025 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:33.849030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:33.856258 (+ 7228us) server_negotiation.cc:407] Connection header received
0929 10:28:33.856434 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:33.856438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:33.856501 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:33.856584 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:33.857735 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.858654 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.860051 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.860429 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.860516 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.863036 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:33.863040 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:33.863043 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:33.863079 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:33.864985 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:33.865539 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:33.865543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:33.865545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:33.865635 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:33.865989 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:33.865993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:33.865994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:33.866272 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:33.866375 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:33.866775 (+ 400us) server_negotiation.cc:299] Negotiation successful
0929 10:28:33.866898 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 10:28:33.874920 27919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.863607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47174 (local address 127.20.92.60:46243)
0929 10:28:33.863727 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:33.863732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:33.864507 (+ 775us) server_negotiation.cc:407] Connection header received
0929 10:28:33.864682 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:33.864686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:33.864733 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:33.864785 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:33.866072 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.866916 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.868072 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.868433 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.868506 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.870372 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:33.870376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:33.870379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:33.870406 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:33.872084 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:33.872653 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:33.872657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:33.872660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:33.872744 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:33.873020 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:33.873027 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:33.873031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:33.873274 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:33.873416 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:33.873735 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 10:28:33.874753 (+ 1018us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 10:28:33.877017 28200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.849122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47164 (local address 127.20.92.60:46243)
0929 10:28:33.849236 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:33.849254 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:33.867167 (+ 17913us) server_negotiation.cc:407] Connection header received
0929 10:28:33.867403 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:33.867406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:33.867459 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:33.867513 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:33.868572 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.869381 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.870520 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.870895 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.870977 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.872672 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:33.872676 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:33.872678 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:33.872704 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:33.874507 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:33.875316 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:33.875321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:33.875323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:33.875411 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:33.875780 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:33.875786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:33.875789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:33.876017 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:33.876149 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:33.876738 (+ 589us) server_negotiation.cc:299] Negotiation successful
0929 10:28:33.876862 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 10:28:33.878998 28197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:33.866340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49198 (local address 127.20.92.61:41373)
0929 10:28:33.866456 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:33.866460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:33.868755 (+ 2295us) server_negotiation.cc:407] Connection header received
0929 10:28:33.868927 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:33.868930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:33.869283 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:33.869339 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:33.870288 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.871773 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.873322 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:33.873662 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:33.873735 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:33.875520 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:33.875525 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:33.875528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:33.875553 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:33.877309 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:33.877740 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:33.877743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:33.877745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:33.877878 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:33.878139 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:33.878144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:33.878145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:33.878323 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:33.878457 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:33.878681 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 10:28:33.878817 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 10:28:34.834614 28250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:34.834014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.20.92.1:50479)
0929 10:28:34.834390 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:34.834487 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":87,"threads_started":1}
W20260929 10:28:34.834903 27913 heartbeater.cc:644] Failed to heartbeat to 127.20.92.62:38271 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
I20260929 10:28:34.835053 28251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:34.834513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.20.92.3:52199)
0929 10:28:34.834866 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:34.834952 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":64,"threads_started":1}
W20260929 10:28:34.835335 28186 heartbeater.cc:644] Failed to heartbeat to 127.20.92.62:38271 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
I20260929 10:28:34.838413 28252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:34.837911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.20.92.2:34039)
0929 10:28:34.838186 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:34.838295 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":75,"threads_started":1}
W20260929 10:28:34.838641 28050 heartbeater.cc:644] Failed to heartbeat to 127.20.92.62:38271 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
I20260929 10:28:35.346235 28253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47d8539835204349b4a1620d4eabe0a7)
I20260929 10:28:35.346436 28253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: false } }
I20260929 10:28:35.346776 28254 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47d8539835204349b4a1620d4eabe0a7)
I20260929 10:28:35.346938 28254 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: false } }
I20260929 10:28:35.347563 28253 leader_election.cc:288] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47d8539835204349b4a1620d4eabe0a7 (127.20.92.62:38271), 47387bc665724e9792490842b4ffc9ae (127.20.92.60:46243)
I20260929 10:28:35.347806 28254 leader_election.cc:288] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47d8539835204349b4a1620d4eabe0a7 (127.20.92.62:38271), 1c21be5e2e22417fa95a888fc5205dca (127.20.92.61:41373)
I20260929 10:28:35.348349 28255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:35.347650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.0.0.1:36840)
0929 10:28:35.348157 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:35.348237 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":69,"threads_started":1}
W20260929 10:28:35.348526 27536 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
I20260929 10:28:35.348722 28256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:35.347877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.0.0.1:36856)
0929 10:28:35.348512 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:35.348585 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":67,"threads_started":1}
W20260929 10:28:35.348951 27718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
I20260929 10:28:35.349779 28256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:35.349537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.0.0.1:36858)
0929 10:28:35.349659 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:35.349702 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 10:28:35.349828 28255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:35.349558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.0.0.1:36872)
0929 10:28:35.349679 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:35.349726 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260929 10:28:35.349916 27536 leader_election.cc:334] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47d8539835204349b4a1620d4eabe0a7 (127.20.92.62:38271): Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
W20260929 10:28:35.349965 27718 leader_election.cc:334] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47d8539835204349b4a1620d4eabe0a7 (127.20.92.62:38271): Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
I20260929 10:28:35.351943 28258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:35.348043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:41373 (local address 127.0.0.1:49210)
0929 10:28:35.348473 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:35.348486 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:35.348567 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:35.348840 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:35.348843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:35.348865 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:35.348984 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:35.348990 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:35.350142 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:35.350146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:35.350975 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:35.350984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:35.351552 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:35.351558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:35.351638 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:35.351657 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 10:28:35.351736 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 10:28:35.351819 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":47,"threads_started":1}
I20260929 10:28:35.351976 28257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:35.347981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:46243 (local address 127.0.0.1:47182)
0929 10:28:35.348222 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:35.348237 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:35.348353 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:35.348902 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:35.348905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:35.348920 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:35.349024 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:35.349031 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:35.350140 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:35.350146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:35.350948 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:35.350957 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:35.351612 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:35.351619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:35.351692 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:35.351706 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 10:28:35.351801 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 10:28:35.351865 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":63,"threads_started":1}
I20260929 10:28:35.351974 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:35.348217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47182 (local address 127.20.92.60:46243)
0929 10:28:35.348677 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:35.348681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:35.348691 (+ 10us) server_negotiation.cc:407] Connection header received
0929 10:28:35.348731 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:35.348736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:35.348788 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:35.348870 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:35.349161 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:35.350000 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:35.351065 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:35.351468 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:35.351594 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:35.351648 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:35.351824 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 10:28:35.351882 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":51,"threads_started":1}
I20260929 10:28:35.351975 28260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:35.348179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49210 (local address 127.20.92.61:41373)
0929 10:28:35.348564 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:35.348568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:35.348583 (+ 15us) server_negotiation.cc:407] Connection header received
0929 10:28:35.348639 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:35.348644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:35.348703 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:35.349093 (+ 390us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:35.349213 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:35.350012 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:35.351081 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:35.351449 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:35.351518 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:35.351603 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:35.351769 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 10:28:35.351855 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":75,"threads_started":1}
I20260929 10:28:35.352543 27746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c21be5e2e22417fa95a888fc5205dca" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "47387bc665724e9792490842b4ffc9ae" is_pre_election: true
I20260929 10:28:35.352761 27746 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c21be5e2e22417fa95a888fc5205dca in term 1.
I20260929 10:28:35.352758 27561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47387bc665724e9792490842b4ffc9ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1c21be5e2e22417fa95a888fc5205dca" is_pre_election: true
I20260929 10:28:35.352933 27561 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47387bc665724e9792490842b4ffc9ae in term 1.
I20260929 10:28:35.353091 27536 leader_election.cc:302] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c21be5e2e22417fa95a888fc5205dca, 47387bc665724e9792490842b4ffc9ae; no voters: 47d8539835204349b4a1620d4eabe0a7
I20260929 10:28:35.353252 27719 leader_election.cc:302] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c21be5e2e22417fa95a888fc5205dca, 47387bc665724e9792490842b4ffc9ae; no voters: 47d8539835204349b4a1620d4eabe0a7
I20260929 10:28:35.353308 28253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 10:28:35.353363 28253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [term 1 FOLLOWER]: Starting leader election (detected failure of leader 47d8539835204349b4a1620d4eabe0a7)
I20260929 10:28:35.353405 28253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [term 1 FOLLOWER]: Advancing to term 2
I20260929 10:28:35.353592 28254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 10:28:35.353665 28254 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [term 1 FOLLOWER]: Starting leader election (detected failure of leader 47d8539835204349b4a1620d4eabe0a7)
I20260929 10:28:35.353729 28254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [term 1 FOLLOWER]: Advancing to term 2
I20260929 10:28:35.354526 28253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: false } }
I20260929 10:28:35.354738 28254 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: false } }
I20260929 10:28:35.354907 28253 leader_election.cc:288] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [CANDIDATE]: Term 2 election: Requested vote from peers 47d8539835204349b4a1620d4eabe0a7 (127.20.92.62:38271), 47387bc665724e9792490842b4ffc9ae (127.20.92.60:46243)
I20260929 10:28:35.355132 28254 leader_election.cc:288] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [CANDIDATE]: Term 2 election: Requested vote from peers 47d8539835204349b4a1620d4eabe0a7 (127.20.92.62:38271), 1c21be5e2e22417fa95a888fc5205dca (127.20.92.61:41373)
I20260929 10:28:35.355201 28257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:35.354999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.0.0.1:36882)
0929 10:28:35.355080 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:35.355136 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260929 10:28:35.355475 28258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:35.355241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.0.0.1:36886)
0929 10:28:35.355369 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:35.355419 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 10:28:35.355479 27561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47387bc665724e9792490842b4ffc9ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1c21be5e2e22417fa95a888fc5205dca"
I20260929 10:28:35.355476 27746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c21be5e2e22417fa95a888fc5205dca" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "47387bc665724e9792490842b4ffc9ae"
I20260929 10:28:35.355640 27561 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 47387bc665724e9792490842b4ffc9ae in current term 2: Already voted for candidate 1c21be5e2e22417fa95a888fc5205dca in this term.
I20260929 10:28:35.355645 27746 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c21be5e2e22417fa95a888fc5205dca in current term 2: Already voted for candidate 47387bc665724e9792490842b4ffc9ae in this term.
I20260929 10:28:35.356101 28257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:35.355663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.0.0.1:36892)
0929 10:28:35.356018 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:35.356057 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305}
I20260929 10:28:35.356285 28258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:35.356098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.0.0.1:36908)
0929 10:28:35.356187 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:35.356231 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 10:28:35.356380 27718 leader_election.cc:334] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 47d8539835204349b4a1620d4eabe0a7 (127.20.92.62:38271): Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
W20260929 10:28:35.356413 27536 leader_election.cc:334] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 47d8539835204349b4a1620d4eabe0a7 (127.20.92.62:38271): Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
I20260929 10:28:35.356444 27718 leader_election.cc:302] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47387bc665724e9792490842b4ffc9ae; no voters: 1c21be5e2e22417fa95a888fc5205dca, 47d8539835204349b4a1620d4eabe0a7
I20260929 10:28:35.356472 27536 leader_election.cc:302] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c21be5e2e22417fa95a888fc5205dca; no voters: 47387bc665724e9792490842b4ffc9ae, 47d8539835204349b4a1620d4eabe0a7
I20260929 10:28:35.356591 28253 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 10:28:35.356591 28254 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 10:28:35.836200 28263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:35.835641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.20.92.1:35895)
0929 10:28:35.835993 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:35.836076 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":103,"threads_started":1}
I20260929 10:28:35.836344 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:35.835871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.20.92.3:58881)
0929 10:28:35.836164 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:35.836221 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":78,"threads_started":1}
I20260929 10:28:35.839624 28265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:35.839194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.20.92.2:35853)
0929 10:28:35.839443 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:35.839511 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":80,"threads_started":1}
I20260929 10:28:36.837546 28266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:36.837084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.20.92.3:60719)
0929 10:28:36.837364 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:36.837433 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":83,"threads_started":1}
I20260929 10:28:36.837646 28267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:36.837139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.20.92.1:39833)
0929 10:28:36.837471 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:36.837530 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":73,"threads_started":1}
I20260929 10:28:36.840883 28268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:36.840379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.20.92.2:35669)
0929 10:28:36.840653 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:36.840736 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":112,"threads_started":1}
I20260929 10:28:37.015713 28269 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:28:37.015852 28269 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: false } }
I20260929 10:28:37.016220 28269 leader_election.cc:288] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 47d8539835204349b4a1620d4eabe0a7 (127.20.92.62:38271), 47387bc665724e9792490842b4ffc9ae (127.20.92.60:46243)
I20260929 10:28:37.016870 28270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:37.016477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.0.0.1:37122)
0929 10:28:37.016698 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:37.016768 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":95,"threads_started":1}
I20260929 10:28:37.017083 27746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c21be5e2e22417fa95a888fc5205dca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "47387bc665724e9792490842b4ffc9ae" is_pre_election: true
I20260929 10:28:37.017243 27746 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c21be5e2e22417fa95a888fc5205dca in term 2.
I20260929 10:28:37.017603 27536 leader_election.cc:302] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c21be5e2e22417fa95a888fc5205dca, 47387bc665724e9792490842b4ffc9ae; no voters:
I20260929 10:28:37.017771 28270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:37.017450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.0.0.1:37130)
0929 10:28:37.017550 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:37.017597 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 10:28:37.017896 28269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260929 10:28:37.017915 27536 leader_election.cc:334] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 47d8539835204349b4a1620d4eabe0a7 (127.20.92.62:38271): Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
I20260929 10:28:37.017974 28269 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:28:37.018020 28269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [term 2 FOLLOWER]: Advancing to term 3
I20260929 10:28:37.019033 28269 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: false } }
I20260929 10:28:37.019361 28269 leader_election.cc:288] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [CANDIDATE]: Term 3 election: Requested vote from peers 47d8539835204349b4a1620d4eabe0a7 (127.20.92.62:38271), 47387bc665724e9792490842b4ffc9ae (127.20.92.60:46243)
I20260929 10:28:37.019906 28270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:37.019635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.0.0.1:37142)
0929 10:28:37.019794 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:37.019854 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 10:28:37.020083 27746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c21be5e2e22417fa95a888fc5205dca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "47387bc665724e9792490842b4ffc9ae"
I20260929 10:28:37.020217 27746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [term 2 FOLLOWER]: Advancing to term 3
I20260929 10:28:37.020537 28270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:37.020357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.0.0.1:37158)
0929 10:28:37.020437 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:37.020485 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260929 10:28:37.020634 27536 leader_election.cc:334] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 47d8539835204349b4a1620d4eabe0a7 (127.20.92.62:38271): Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
I20260929 10:28:37.021332 27746 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c21be5e2e22417fa95a888fc5205dca in term 3.
I20260929 10:28:37.021685 27536 leader_election.cc:302] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c21be5e2e22417fa95a888fc5205dca, 47387bc665724e9792490842b4ffc9ae; no voters: 47d8539835204349b4a1620d4eabe0a7
I20260929 10:28:37.021907 28269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [term 3 FOLLOWER]: Leader election won for term 3
I20260929 10:28:37.022099 28269 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [term 3 LEADER]: Becoming Leader. State: Replica: 1c21be5e2e22417fa95a888fc5205dca, State: Running, Role: LEADER
I20260929 10:28:37.022444 28269 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: false } }
I20260929 10:28:37.024008 28272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c21be5e2e22417fa95a888fc5205dca. Latest consensus state: current_term: 3 leader_uuid: "1c21be5e2e22417fa95a888fc5205dca" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: false } } }
I20260929 10:28:37.024238 28272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:37.024555 28274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:28:37.027063 28274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:28:37.027559 28274 catalog_manager.cc:1259] Loaded cluster ID: e20e448549d846c1b8d5f05daf6c93da
I20260929 10:28:37.027621 28274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:28:37.028268 28274 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:28:37.028702 28274 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca: Loaded TSK: 0
I20260929 10:28:37.029479 28274 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:28:37.105382 27550 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:28:37.108516 27550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:28:37.111573 28270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:37.111309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.0.0.1:37178)
0929 10:28:37.111448 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:37.111502 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 10:28:37.111558 27746 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [term 3 FOLLOWER]: Refusing update from remote peer 1c21be5e2e22417fa95a888fc5205dca: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260929 10:28:37.112097 28272 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [LEADER]: Connected to new peer: Peer: permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260929 10:28:37.112607 28270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:37.112333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.0.0.1:37184)
0929 10:28:37.112479 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:37.112549 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 10:28:37.112860 27536 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca -> Peer 47d8539835204349b4a1620d4eabe0a7 (127.20.92.62:38271): Couldn't send request to peer 47d8539835204349b4a1620d4eabe0a7. Status: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 10:28:37.114028 28276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c21be5e2e22417fa95a888fc5205dca. Latest consensus state: current_term: 3 leader_uuid: "1c21be5e2e22417fa95a888fc5205dca" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: false } } }
I20260929 10:28:37.114189 28276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:28:37.115489 28272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "1c21be5e2e22417fa95a888fc5205dca" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: false } } }
I20260929 10:28:37.115608 28272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:37.115754 28276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "1c21be5e2e22417fa95a888fc5205dca" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d8539835204349b4a1620d4eabe0a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38271 } } peers { permanent_uuid: "1c21be5e2e22417fa95a888fc5205dca" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 41373 } attrs { promote: false } } peers { permanent_uuid: "47387bc665724e9792490842b4ffc9ae" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 46243 } attrs { promote: false } } }
I20260929 10:28:37.115903 28276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47387bc665724e9792490842b4ffc9ae [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:28:37.127005 28270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:37.122107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:35207 (local address 127.0.0.1:49628)
0929 10:28:37.122469 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:37.122484 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:37.122599 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:37.122936 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:37.122942 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:37.122965 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:37.123104 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:37.123112 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:37.124508 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:37.124511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:37.125517 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:37.125526 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:37.126590 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:37.126593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:37.126690 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:37.126721 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 10:28:37.126773 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 10:28:37.126880 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":71}
I20260929 10:28:37.127557 28283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:37.122466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52606 (local address 127.20.92.3:40193)
0929 10:28:37.122726 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:37.122733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:37.123126 (+ 393us) server_negotiation.cc:407] Connection header received
0929 10:28:37.123199 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:37.123202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:37.123258 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:37.123333 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:37.123661 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:37.124432 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:37.126004 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:37.126620 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:37.126714 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:37.127295 (+ 581us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:37.127337 (+ 42us) server_negotiation.cc:299] Negotiation successful
0929 10:28:37.127426 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":100,"threads_started":1}
I20260929 10:28:37.127565 28281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:37.122219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49628 (local address 127.20.92.1:35207)
0929 10:28:37.122655 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:37.122661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:37.122678 (+ 17us) server_negotiation.cc:407] Connection header received
0929 10:28:37.122746 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:37.122748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:37.122801 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:37.122899 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:37.123233 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:37.124390 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:37.125639 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:37.126441 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:37.127220 (+ 779us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:37.127303 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:37.127340 (+ 37us) server_negotiation.cc:299] Negotiation successful
0929 10:28:37.127426 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":96,"threads_started":1}
I20260929 10:28:37.127703 28282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:37.122344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:40193 (local address 127.0.0.1:52606)
0929 10:28:37.122937 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:37.122951 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:37.123039 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:37.123433 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:37.123436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:37.123452 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:37.123536 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:37.123541 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:37.124549 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:37.124552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:37.125863 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:37.125873 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:37.126831 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:37.126834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:37.126922 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:37.126944 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 10:28:37.127001 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 10:28:37.127064 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":3,"thread_start_us":93,"threads_started":1}
I20260929 10:28:37.127916 28284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:37.122883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:39893 (local address 127.0.0.1:55170)
0929 10:28:37.123506 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:37.123516 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:37.123626 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:37.123931 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:37.123933 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:37.123997 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:37.124086 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:37.124091 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:37.125065 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:37.125068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:37.126531 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:37.126540 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:37.127501 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:37.127505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:37.127598 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:37.127623 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 10:28:37.127697 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 10:28:37.127791 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":88,"threads_started":1}
I20260929 10:28:37.128089 27846 tablet_service.cc:1467] Processing CreateTablet for tablet 7bb639e7e5da49eaab27a2ccd7a6e759 (DEFAULT_TABLE table=test-table [id=0db5feff570746b29b8af016a7d80c24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:37.128193 28119 tablet_service.cc:1467] Processing CreateTablet for tablet 7bb639e7e5da49eaab27a2ccd7a6e759 (DEFAULT_TABLE table=test-table [id=0db5feff570746b29b8af016a7d80c24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:37.128280 28285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:37.123151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55170 (local address 127.20.92.2:39893)
0929 10:28:37.123372 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:37.123377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:37.123593 (+ 216us) server_negotiation.cc:407] Connection header received
0929 10:28:37.123760 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:37.123767 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:37.123821 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:37.123906 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:37.124210 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:37.124935 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:37.126665 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:37.127272 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:37.127389 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:37.127446 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:37.128073 (+ 627us) server_negotiation.cc:299] Negotiation successful
0929 10:28:37.128151 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":78,"threads_started":1}
I20260929 10:28:37.128412 27846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bb639e7e5da49eaab27a2ccd7a6e759. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:37.128485 28119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bb639e7e5da49eaab27a2ccd7a6e759. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:37.128929 27983 tablet_service.cc:1467] Processing CreateTablet for tablet 7bb639e7e5da49eaab27a2ccd7a6e759 (DEFAULT_TABLE table=test-table [id=0db5feff570746b29b8af016a7d80c24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:37.129179 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bb639e7e5da49eaab27a2ccd7a6e759. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:37.130630 28286 tablet_bootstrap.cc:492] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 77ff8f0666ff4666b205c8addfa5706f: Bootstrap starting.
I20260929 10:28:37.131387 28287 tablet_bootstrap.cc:492] T 7bb639e7e5da49eaab27a2ccd7a6e759 P b2c6034747b64db3910457be844cc69b: Bootstrap starting.
I20260929 10:28:37.131467 28286 tablet_bootstrap.cc:654] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 77ff8f0666ff4666b205c8addfa5706f: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:37.131744 28288 tablet_bootstrap.cc:492] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 52a6fe68df7942e4bd2675a0075bac88: Bootstrap starting.
I20260929 10:28:37.132265 28287 tablet_bootstrap.cc:654] T 7bb639e7e5da49eaab27a2ccd7a6e759 P b2c6034747b64db3910457be844cc69b: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:37.132522 28288 tablet_bootstrap.cc:654] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 52a6fe68df7942e4bd2675a0075bac88: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:37.133466 28287 tablet_bootstrap.cc:492] T 7bb639e7e5da49eaab27a2ccd7a6e759 P b2c6034747b64db3910457be844cc69b: No bootstrap required, opened a new log
I20260929 10:28:37.133567 28287 ts_tablet_manager.cc:1397] T 7bb639e7e5da49eaab27a2ccd7a6e759 P b2c6034747b64db3910457be844cc69b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:28:37.133677 28286 tablet_bootstrap.cc:492] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 77ff8f0666ff4666b205c8addfa5706f: No bootstrap required, opened a new log
I20260929 10:28:37.133814 28286 ts_tablet_manager.cc:1397] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 77ff8f0666ff4666b205c8addfa5706f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 10:28:37.134066 28287 raft_consensus.cc:348] T 7bb639e7e5da49eaab27a2ccd7a6e759 P b2c6034747b64db3910457be844cc69b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c6034747b64db3910457be844cc69b" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35207 } } peers { permanent_uuid: "77ff8f0666ff4666b205c8addfa5706f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40193 } } peers { permanent_uuid: "52a6fe68df7942e4bd2675a0075bac88" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39893 } }
I20260929 10:28:37.134230 28287 raft_consensus.cc:374] T 7bb639e7e5da49eaab27a2ccd7a6e759 P b2c6034747b64db3910457be844cc69b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:37.134290 28287 raft_consensus.cc:729] T 7bb639e7e5da49eaab27a2ccd7a6e759 P b2c6034747b64db3910457be844cc69b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2c6034747b64db3910457be844cc69b, State: Initialized, Role: FOLLOWER
I20260929 10:28:37.134230 28286 raft_consensus.cc:348] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 77ff8f0666ff4666b205c8addfa5706f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c6034747b64db3910457be844cc69b" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35207 } } peers { permanent_uuid: "77ff8f0666ff4666b205c8addfa5706f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40193 } } peers { permanent_uuid: "52a6fe68df7942e4bd2675a0075bac88" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39893 } }
I20260929 10:28:37.134334 28286 raft_consensus.cc:374] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 77ff8f0666ff4666b205c8addfa5706f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:37.134413 28286 raft_consensus.cc:729] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 77ff8f0666ff4666b205c8addfa5706f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ff8f0666ff4666b205c8addfa5706f, State: Initialized, Role: FOLLOWER
I20260929 10:28:37.134507 28288 tablet_bootstrap.cc:492] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 52a6fe68df7942e4bd2675a0075bac88: No bootstrap required, opened a new log
I20260929 10:28:37.134473 28287 consensus_queue.cc:260] T 7bb639e7e5da49eaab27a2ccd7a6e759 P b2c6034747b64db3910457be844cc69b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c6034747b64db3910457be844cc69b" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35207 } } peers { permanent_uuid: "77ff8f0666ff4666b205c8addfa5706f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40193 } } peers { permanent_uuid: "52a6fe68df7942e4bd2675a0075bac88" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39893 } }
I20260929 10:28:37.134593 28288 ts_tablet_manager.cc:1397] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 52a6fe68df7942e4bd2675a0075bac88: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 10:28:37.134547 28286 consensus_queue.cc:260] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 77ff8f0666ff4666b205c8addfa5706f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c6034747b64db3910457be844cc69b" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35207 } } peers { permanent_uuid: "77ff8f0666ff4666b205c8addfa5706f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40193 } } peers { permanent_uuid: "52a6fe68df7942e4bd2675a0075bac88" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39893 } }
I20260929 10:28:37.134896 28287 ts_tablet_manager.cc:1428] T 7bb639e7e5da49eaab27a2ccd7a6e759 P b2c6034747b64db3910457be844cc69b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 10:28:37.134927 28286 ts_tablet_manager.cc:1428] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 77ff8f0666ff4666b205c8addfa5706f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:28:37.135010 28288 raft_consensus.cc:348] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 52a6fe68df7942e4bd2675a0075bac88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c6034747b64db3910457be844cc69b" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35207 } } peers { permanent_uuid: "77ff8f0666ff4666b205c8addfa5706f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40193 } } peers { permanent_uuid: "52a6fe68df7942e4bd2675a0075bac88" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39893 } }
I20260929 10:28:37.135104 28288 raft_consensus.cc:374] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 52a6fe68df7942e4bd2675a0075bac88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:37.135139 28288 raft_consensus.cc:729] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 52a6fe68df7942e4bd2675a0075bac88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a6fe68df7942e4bd2675a0075bac88, State: Initialized, Role: FOLLOWER
I20260929 10:28:37.135263 28288 consensus_queue.cc:260] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 52a6fe68df7942e4bd2675a0075bac88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c6034747b64db3910457be844cc69b" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35207 } } peers { permanent_uuid: "77ff8f0666ff4666b205c8addfa5706f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40193 } } peers { permanent_uuid: "52a6fe68df7942e4bd2675a0075bac88" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39893 } }
I20260929 10:28:37.135623 28288 ts_tablet_manager.cc:1428] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 52a6fe68df7942e4bd2675a0075bac88: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:28:37.135859 28267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:37.135320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.20.92.1:43875)
0929 10:28:37.135694 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:37.135768 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282}
I20260929 10:28:37.136077 28266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:37.135829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.20.92.3:51497)
0929 10:28:37.135942 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:37.136016 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 10:28:37.136859 28268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:37.136608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.20.92.2:52597)
0929 10:28:37.136725 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:37.136794 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 10:28:37.138434 28266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:37.138085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.20.92.3:48885)
0929 10:28:37.138324 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:37.138373 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260929 10:28:37.139345 28187 heartbeater.cc:503] Master 127.20.92.61:41373 was elected leader, sending a full tablet report...
I20260929 10:28:37.139909 28267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:37.139559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.20.92.1:55827)
0929 10:28:37.139784 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:37.139841 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260929 10:28:37.140219 27914 heartbeater.cc:503] Master 127.20.92.61:41373 was elected leader, sending a full tablet report...
I20260929 10:28:37.265246 28293 raft_consensus.cc:482] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 77ff8f0666ff4666b205c8addfa5706f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:28:37.265421 28293 raft_consensus.cc:504] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 77ff8f0666ff4666b205c8addfa5706f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c6034747b64db3910457be844cc69b" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35207 } } peers { permanent_uuid: "77ff8f0666ff4666b205c8addfa5706f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40193 } } peers { permanent_uuid: "52a6fe68df7942e4bd2675a0075bac88" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39893 } }
I20260929 10:28:37.266669 28293 leader_election.cc:288] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 77ff8f0666ff4666b205c8addfa5706f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2c6034747b64db3910457be844cc69b (127.20.92.1:35207), 52a6fe68df7942e4bd2675a0075bac88 (127.20.92.2:39893)
I20260929 10:28:37.272538 28281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:37.267164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:49605 (local address 127.20.92.1:35207)
0929 10:28:37.267279 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:37.267285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:37.267397 (+ 112us) server_negotiation.cc:407] Connection header received
0929 10:28:37.267542 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:37.267544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:37.267631 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:37.267731 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:37.268243 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:37.269113 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:37.271079 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:37.271790 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:37.271917 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:37.271990 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:37.272321 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 10:28:37.272404 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 10:28:37.272538 28285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:37.267176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:36363 (local address 127.20.92.2:39893)
0929 10:28:37.267328 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:37.267333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:37.267592 (+ 259us) server_negotiation.cc:407] Connection header received
0929 10:28:37.267659 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:37.267666 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:37.267729 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:37.267805 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:37.268243 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:37.269157 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:37.271079 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:37.271789 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:37.271917 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:37.271990 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:37.272321 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 10:28:37.272404 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 10:28:37.272534 28266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:37.267063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:35207 (local address 127.20.92.3:49605)
0929 10:28:37.267267 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:37.267282 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:37.267420 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:37.267806 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:37.267812 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:37.267837 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:37.268088 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:37.268093 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:37.269300 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:37.269304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:37.270908 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:37.270920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:37.271943 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:37.271947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:37.272119 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:37.272169 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 10:28:37.272254 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 10:28:37.272347 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":338,"spinlock_wait_cycles":2176}
I20260929 10:28:37.272557 28295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:37.267068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:39893 (local address 127.20.92.3:36363)
0929 10:28:37.267484 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:37.267498 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:37.267583 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:37.267846 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:37.267848 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:37.267859 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:37.268085 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:37.268093 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:37.269300 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:37.269304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:37.270933 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:37.270939 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:37.271947 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:37.271950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:37.272114 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:37.272173 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 10:28:37.272265 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 10:28:37.272347 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":305,"thread_start_us":125,"threads_started":1}
I20260929 10:28:37.273262 27866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bb639e7e5da49eaab27a2ccd7a6e759" candidate_uuid: "77ff8f0666ff4666b205c8addfa5706f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2c6034747b64db3910457be844cc69b" is_pre_election: true
I20260929 10:28:37.273348 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bb639e7e5da49eaab27a2ccd7a6e759" candidate_uuid: "77ff8f0666ff4666b205c8addfa5706f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a6fe68df7942e4bd2675a0075bac88" is_pre_election: true
I20260929 10:28:37.273438 27866 raft_consensus.cc:2454] T 7bb639e7e5da49eaab27a2ccd7a6e759 P b2c6034747b64db3910457be844cc69b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77ff8f0666ff4666b205c8addfa5706f in term 0.
I20260929 10:28:37.273559 28003 raft_consensus.cc:2454] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 52a6fe68df7942e4bd2675a0075bac88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77ff8f0666ff4666b205c8addfa5706f in term 0.
I20260929 10:28:37.273972 28074 leader_election.cc:302] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 77ff8f0666ff4666b205c8addfa5706f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 52a6fe68df7942e4bd2675a0075bac88, 77ff8f0666ff4666b205c8addfa5706f, b2c6034747b64db3910457be844cc69b; no voters:
I20260929 10:28:37.274300 28293 raft_consensus.cc:2781] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 77ff8f0666ff4666b205c8addfa5706f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:28:37.274365 28293 raft_consensus.cc:482] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 77ff8f0666ff4666b205c8addfa5706f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:28:37.274403 28293 raft_consensus.cc:3037] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 77ff8f0666ff4666b205c8addfa5706f [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:37.275522 28293 raft_consensus.cc:504] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 77ff8f0666ff4666b205c8addfa5706f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c6034747b64db3910457be844cc69b" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35207 } } peers { permanent_uuid: "77ff8f0666ff4666b205c8addfa5706f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40193 } } peers { permanent_uuid: "52a6fe68df7942e4bd2675a0075bac88" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39893 } }
I20260929 10:28:37.275898 28293 leader_election.cc:288] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 77ff8f0666ff4666b205c8addfa5706f [CANDIDATE]: Term 1 election: Requested vote from peers b2c6034747b64db3910457be844cc69b (127.20.92.1:35207), 52a6fe68df7942e4bd2675a0075bac88 (127.20.92.2:39893)
I20260929 10:28:37.276265 27866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bb639e7e5da49eaab27a2ccd7a6e759" candidate_uuid: "77ff8f0666ff4666b205c8addfa5706f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2c6034747b64db3910457be844cc69b"
I20260929 10:28:37.276309 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bb639e7e5da49eaab27a2ccd7a6e759" candidate_uuid: "77ff8f0666ff4666b205c8addfa5706f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a6fe68df7942e4bd2675a0075bac88"
I20260929 10:28:37.276413 28003 raft_consensus.cc:3037] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 52a6fe68df7942e4bd2675a0075bac88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:37.276410 27866 raft_consensus.cc:3037] T 7bb639e7e5da49eaab27a2ccd7a6e759 P b2c6034747b64db3910457be844cc69b [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:37.277540 27866 raft_consensus.cc:2454] T 7bb639e7e5da49eaab27a2ccd7a6e759 P b2c6034747b64db3910457be844cc69b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77ff8f0666ff4666b205c8addfa5706f in term 1.
I20260929 10:28:37.277539 28003 raft_consensus.cc:2454] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 52a6fe68df7942e4bd2675a0075bac88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77ff8f0666ff4666b205c8addfa5706f in term 1.
I20260929 10:28:37.277891 28074 leader_election.cc:302] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 77ff8f0666ff4666b205c8addfa5706f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77ff8f0666ff4666b205c8addfa5706f, b2c6034747b64db3910457be844cc69b; no voters:
I20260929 10:28:37.278064 28293 raft_consensus.cc:2781] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 77ff8f0666ff4666b205c8addfa5706f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:37.278314 28293 raft_consensus.cc:686] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 77ff8f0666ff4666b205c8addfa5706f [term 1 LEADER]: Becoming Leader. State: Replica: 77ff8f0666ff4666b205c8addfa5706f, State: Running, Role: LEADER
I20260929 10:28:37.278776 28293 consensus_queue.cc:237] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 77ff8f0666ff4666b205c8addfa5706f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c6034747b64db3910457be844cc69b" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35207 } } peers { permanent_uuid: "77ff8f0666ff4666b205c8addfa5706f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40193 } } peers { permanent_uuid: "52a6fe68df7942e4bd2675a0075bac88" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39893 } }
I20260929 10:28:37.279057 28295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:37.278805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.20.92.3:46875)
0929 10:28:37.278919 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:37.278991 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 10:28:37.281260 27549 catalog_manager.cc:5495] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 77ff8f0666ff4666b205c8addfa5706f reported cstate change: term changed from 0 to 1, leader changed from <none> to 77ff8f0666ff4666b205c8addfa5706f (127.20.92.3). New cstate: current_term: 1 leader_uuid: "77ff8f0666ff4666b205c8addfa5706f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c6034747b64db3910457be844cc69b" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77ff8f0666ff4666b205c8addfa5706f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52a6fe68df7942e4bd2675a0075bac88" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39893 } health_report { overall_health: UNKNOWN } } }
I20260929 10:28:37.326215 28283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:37.322394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52620 (local address 127.20.92.3:40193)
0929 10:28:37.322533 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:37.322539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:37.322554 (+ 15us) server_negotiation.cc:407] Connection header received
0929 10:28:37.322641 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:37.322644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:37.322709 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:37.322823 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:28:37.323069 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:37.323817 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:37.324679 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:37.324921 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:37.325034 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:37.325328 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:28:37.325807 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:28:37.325933 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:37.326006 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 10:28:37.326065 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 10:28:37.329465 27866 raft_consensus.cc:1261] T 7bb639e7e5da49eaab27a2ccd7a6e759 P b2c6034747b64db3910457be844cc69b [term 1 FOLLOWER]: Refusing update from remote peer 77ff8f0666ff4666b205c8addfa5706f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:37.329676 28003 raft_consensus.cc:1261] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 52a6fe68df7942e4bd2675a0075bac88 [term 1 FOLLOWER]: Refusing update from remote peer 77ff8f0666ff4666b205c8addfa5706f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:37.330117 28296 consensus_queue.cc:1035] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 77ff8f0666ff4666b205c8addfa5706f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2c6034747b64db3910457be844cc69b" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:28:37.330320 28293 consensus_queue.cc:1035] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 77ff8f0666ff4666b205c8addfa5706f [LEADER]: Connected to new peer: Peer: permanent_uuid: "52a6fe68df7942e4bd2675a0075bac88" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:28:37.332571 28267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:37.332173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.20.92.1:53055)
0929 10:28:37.332415 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:37.332483 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260929 10:28:37.332803 28051 heartbeater.cc:503] Master 127.20.92.61:41373 was elected leader, sending a full tablet report...
I20260929 10:28:37.333460 28268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:37.333170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.20.92.2:35135)
0929 10:28:37.333330 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:37.333383 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 10:28:37.335891 28295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:37.335608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.20.92.3:38937)
0929 10:28:37.335733 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:37.335805 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 10:28:37.335935 28302 mvcc.cc:205] Tried to move back new op lower bound from 7334615930175733760 to 7334615929976750080. Current Snapshot: MvccSnapshot[applied={T|T < 7334615930175733760 or (T in {7334615930175733760})}]
I20260929 10:28:37.336586 28268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:37.336265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.20.92.2:34881)
0929 10:28:37.336468 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:37.336521 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260929 10:28:37.337371 28267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:37.336982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38271 (local address 127.20.92.1:33323)
0929 10:28:37.337234 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:37.337286 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
I20260929 10:28:37.346007 27551 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:28:37.346166 27551 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:28:37.350508 27551 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c21be5e2e22417fa95a888fc5205dca: Sending DeleteTablet for 3 replicas of tablet 7bb639e7e5da49eaab27a2ccd7a6e759
I20260929 10:28:37.351406 27983 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bb639e7e5da49eaab27a2ccd7a6e759 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:37 UTC) from {username='oryx', principal='oryx/127.20.92.61@KRBTEST.COM'} at 127.0.0.1:55170
I20260929 10:28:37.351620 27846 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bb639e7e5da49eaab27a2ccd7a6e759 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:37 UTC) from {username='oryx', principal='oryx/127.20.92.61@KRBTEST.COM'} at 127.0.0.1:49628
I20260929 10:28:37.351724 28119 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bb639e7e5da49eaab27a2ccd7a6e759 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:37 UTC) from {username='oryx', principal='oryx/127.20.92.61@KRBTEST.COM'} at 127.0.0.1:52606
I20260929 10:28:37.352609 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 27695
I20260929 10:28:37.351995 28310 tablet_replica.cc:330] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 52a6fe68df7942e4bd2675a0075bac88: stopping tablet replica
I20260929 10:28:37.355355 28310 raft_consensus.cc:2229] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 52a6fe68df7942e4bd2675a0075bac88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:28:37.355520 28310 raft_consensus.cc:2258] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 52a6fe68df7942e4bd2675a0075bac88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:37.356374 28310 ts_tablet_manager.cc:1905] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 52a6fe68df7942e4bd2675a0075bac88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:37.357988 28312 tablet_replica.cc:330] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 77ff8f0666ff4666b205c8addfa5706f: stopping tablet replica
I20260929 10:28:37.358192 28312 raft_consensus.cc:2229] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 77ff8f0666ff4666b205c8addfa5706f [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:28:37.358642 28312 raft_consensus.cc:2258] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 77ff8f0666ff4666b205c8addfa5706f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:37.359319 28310 ts_tablet_manager.cc:1918] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 52a6fe68df7942e4bd2675a0075bac88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:28:37.359534 28312 ts_tablet_manager.cc:1905] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 77ff8f0666ff4666b205c8addfa5706f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:37.359608 28310 log.cc:1196] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 52a6fe68df7942e4bd2675a0075bac88: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-1/wal/wals/7bb639e7e5da49eaab27a2ccd7a6e759
I20260929 10:28:37.360044 28310 ts_tablet_manager.cc:1939] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 52a6fe68df7942e4bd2675a0075bac88: Deleting consensus metadata
I20260929 10:28:37.361023 27535 catalog_manager.cc:4841] TS 52a6fe68df7942e4bd2675a0075bac88 (127.20.92.2:39893): tablet 7bb639e7e5da49eaab27a2ccd7a6e759 (table test-table [id=0db5feff570746b29b8af016a7d80c24]) successfully deleted
I20260929 10:28:37.361580 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 27920
I20260929 10:28:37.361893 28282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:37.361366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:35207 (local address 127.0.0.1:49632)
0929 10:28:37.361716 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:37.361796 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.1:35207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247}
W20260929 10:28:37.363221 27537 catalog_manager.cc:4584] TS b2c6034747b64db3910457be844cc69b (127.20.92.1:35207): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7bb639e7e5da49eaab27a2ccd7a6e759: Network error: Client connection negotiation failed: client connection to 127.20.92.1:35207: connect: Connection refused (error 111)
I20260929 10:28:37.363708 28312 ts_tablet_manager.cc:1918] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 77ff8f0666ff4666b205c8addfa5706f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:28:37.363811 28312 log.cc:1196] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 77ff8f0666ff4666b205c8addfa5706f: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677679287822-20848-0/minicluster-data/ts-2/wal/wals/7bb639e7e5da49eaab27a2ccd7a6e759
I20260929 10:28:37.364107 28312 ts_tablet_manager.cc:1939] T 7bb639e7e5da49eaab27a2ccd7a6e759 P 77ff8f0666ff4666b205c8addfa5706f: Deleting consensus metadata
I20260929 10:28:37.365113 27537 catalog_manager.cc:4841] TS 77ff8f0666ff4666b205c8addfa5706f (127.20.92.3:40193): tablet 7bb639e7e5da49eaab27a2ccd7a6e759 (table test-table [id=0db5feff570746b29b8af016a7d80c24]) successfully deleted
I20260929 10:28:37.370251 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 28056
I20260929 10:28:37.376482 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 27413
I20260929 10:28:37.382938 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 27522
2026-09-29T10:28:37Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7532 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:28:37 dist-test-slave-9cj5 krb5kdc[28315](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:28:37 dist-test-slave-9cj5 krb5kdc[28315](info): set up 2 sockets
Sep 29 10:28:37 dist-test-slave-9cj5 krb5kdc[28315](info): commencing operation
krb5kdc: starting...
W20260929 10:28:38.455160 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:28:38 dist-test-slave-9cj5 krb5kdc[28315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:28:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:28:38Z Disabled control of system clock
WARNING: no policy specified for oryx/127.20.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.62.keytab.
Entry for principal oryx/127.20.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.62.keytab.
I20260929 10:28:38.581537 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:39129
--webserver_interface=127.20.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.62.keytab
--principal=oryx/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:36223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:39129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:38.696579 28328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:38.696904 28328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:38.696973 28328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:38.701056 28328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:28:38.701169 28328 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 10:28:38.701197 28328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:38.701232 28328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:38.701269 28328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:28:38.701304 28328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:28:38.706923 28328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:36223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:39129
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.62.keytab
--principal=oryx/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:39129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:38.708101 28328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:38.709160 28328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:38.717701 28333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:38.717859 28334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:38.717793 28336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:38.718140 28328 server_base.cc:1034] running on GCE node
I20260929 10:28:38.718717 28328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:38.719848 28328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:38.721050 28328 hybrid_clock.cc:648] HybridClock initialized: now 1790677718721025 us; error 36 us; skew 500 ppm
Sep 29 10:28:38 dist-test-slave-9cj5 krb5kdc[28315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677718, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:38.723826 28328 init.cc:377] Logged in from keytab as oryx/127.20.92.62@KRBTEST.COM (short username oryx)
I20260929 10:28:38.724737 28328 webserver.cc:460] Webserver started at http://127.20.92.62:34025/ using document root <none> and password file <none>
I20260929 10:28:38.725317 28328 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:38.725410 28328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:38.725680 28328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:38.727860 28328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "d4bcb22f0ca84d1e92a09724ab98d198"
format_stamp: "Formatted at 2026-09-29 10:28:38 on dist-test-slave-9cj5"
server_key: "70fc0274402c8f8f587af56a86331745"
server_key_iv: "faf5b591886062a43139a9e4b46e89bc"
server_key_version: "encryptionkey@0"
I20260929 10:28:38.728448 28328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "d4bcb22f0ca84d1e92a09724ab98d198"
format_stamp: "Formatted at 2026-09-29 10:28:38 on dist-test-slave-9cj5"
server_key: "70fc0274402c8f8f587af56a86331745"
server_key_iv: "faf5b591886062a43139a9e4b46e89bc"
server_key_version: "encryptionkey@0"
I20260929 10:28:38.732437 28328 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260929 10:28:38.735069 28343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:38.736614 28328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 10:28:38.736838 28328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "d4bcb22f0ca84d1e92a09724ab98d198"
format_stamp: "Formatted at 2026-09-29 10:28:38 on dist-test-slave-9cj5"
server_key: "70fc0274402c8f8f587af56a86331745"
server_key_iv: "faf5b591886062a43139a9e4b46e89bc"
server_key_version: "encryptionkey@0"
I20260929 10:28:38.736977 28328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:38.753815 28328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:38.757295 28328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:38.757539 28328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:38.765645 28328 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:39129
I20260929 10:28:38.765671 28405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:39129 every 8 connection(s)
I20260929 10:28:38.766681 28328 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:28:38.768252 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 28328
I20260929 10:28:38.768405 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:28:38.768729 20848 external_mini_cluster.cc:1428] Setting key 5ad6285e6a06a5a57250df40ac193d6f
I20260929 10:28:38.769972 28406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:38.776722 28406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4bcb22f0ca84d1e92a09724ab98d198: Bootstrap starting.
Sep 29 10:28:38 dist-test-slave-9cj5 krb5kdc[28315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM
I20260929 10:28:38.779390 28406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4bcb22f0ca84d1e92a09724ab98d198: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:38.780781 28406 log.cc:824] T 00000000000000000000000000000000 P d4bcb22f0ca84d1e92a09724ab98d198: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:38.783378 28406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4bcb22f0ca84d1e92a09724ab98d198: No bootstrap required, opened a new log
I20260929 10:28:38.784816 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:38.770320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41994 (local address 127.20.92.62:39129)
0929 10:28:38.770728 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:38.770746 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:38.770787 (+ 41us) server_negotiation.cc:407] Connection header received
0929 10:28:38.771637 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:38.771662 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:38.772078 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:38.772457 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:38.773142 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:38.774666 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:38.775570 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:38.776245 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:38.776417 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:38.779429 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:38.779437 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:38.779458 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:38.779492 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:38.781539 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:38.782122 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:38.782127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:38.782133 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:38.782266 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:38.782580 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:38.782582 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:38.782584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:38.783080 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:38.783279 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:38.783586 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 10:28:38.783893 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":113,"threads_started":1}
I20260929 10:28:38.787101 28406 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4bcb22f0ca84d1e92a09724ab98d198 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4bcb22f0ca84d1e92a09724ab98d198" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 39129 } }
I20260929 10:28:38.787354 28406 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4bcb22f0ca84d1e92a09724ab98d198 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:38.787413 28406 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4bcb22f0ca84d1e92a09724ab98d198 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4bcb22f0ca84d1e92a09724ab98d198, State: Initialized, Role: FOLLOWER
I20260929 10:28:38.787765 28406 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4bcb22f0ca84d1e92a09724ab98d198 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4bcb22f0ca84d1e92a09724ab98d198" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 39129 } }
I20260929 10:28:38.787878 28406 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4bcb22f0ca84d1e92a09724ab98d198 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:28:38.787916 28406 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4bcb22f0ca84d1e92a09724ab98d198 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:28:38.787985 28406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4bcb22f0ca84d1e92a09724ab98d198 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:38.788817 28406 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4bcb22f0ca84d1e92a09724ab98d198 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4bcb22f0ca84d1e92a09724ab98d198" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 39129 } }
I20260929 10:28:38.789110 28406 leader_election.cc:302] T 00000000000000000000000000000000 P d4bcb22f0ca84d1e92a09724ab98d198 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4bcb22f0ca84d1e92a09724ab98d198; no voters:
I20260929 10:28:38.789378 28406 leader_election.cc:288] T 00000000000000000000000000000000 P d4bcb22f0ca84d1e92a09724ab98d198 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:28:38.789530 28411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4bcb22f0ca84d1e92a09724ab98d198 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:38.789875 28411 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4bcb22f0ca84d1e92a09724ab98d198 [term 1 LEADER]: Becoming Leader. State: Replica: d4bcb22f0ca84d1e92a09724ab98d198, State: Running, Role: LEADER
I20260929 10:28:38.790174 28411 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4bcb22f0ca84d1e92a09724ab98d198 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4bcb22f0ca84d1e92a09724ab98d198" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 39129 } }
I20260929 10:28:38.790591 28406 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4bcb22f0ca84d1e92a09724ab98d198 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:28:38.792037 28413 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4bcb22f0ca84d1e92a09724ab98d198 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4bcb22f0ca84d1e92a09724ab98d198. Latest consensus state: current_term: 1 leader_uuid: "d4bcb22f0ca84d1e92a09724ab98d198" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4bcb22f0ca84d1e92a09724ab98d198" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 39129 } } }
I20260929 10:28:38.792160 28413 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4bcb22f0ca84d1e92a09724ab98d198 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:38.792501 28412 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4bcb22f0ca84d1e92a09724ab98d198 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4bcb22f0ca84d1e92a09724ab98d198" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4bcb22f0ca84d1e92a09724ab98d198" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 39129 } } }
I20260929 10:28:38.792603 28412 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4bcb22f0ca84d1e92a09724ab98d198 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:38.792558 28422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:28:38.795341 28422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:28:38.800880 28422 catalog_manager.cc:1347] Generated new cluster ID: 029c53a3b15a4f6aa4feaf26730d806a
I20260929 10:28:38.800962 28422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:28:38.810793 28422 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:28:38.812028 28422 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:28:38.816596 28422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4bcb22f0ca84d1e92a09724ab98d198: Generated new TSK 0
I20260929 10:28:38.817355 28422 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.20.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.1.keytab.
Entry for principal oryx/127.20.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.1.keytab.
I20260929 10:28:38.858400 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:0
--local_ip_for_outbound_sockets=127.20.92.1
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:39129
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.1.keytab
--principal=oryx/127.20.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:36223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:38.974128 28432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:38.974439 28432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:38.974519 28432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:38.978626 28432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:38.978729 28432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:38.978808 28432 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 10:28:38.978888 28432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:28:38.984184 28432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:36223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.1.keytab
--principal=oryx/127.20.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.92.62:39129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:38.985304 28432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:38.986374 28432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:38.997028 28438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:38.997028 28440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:38.997028 28437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:38.997397 28432 server_base.cc:1034] running on GCE node
I20260929 10:28:38.997882 28432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:38.998576 28432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:38.999776 28432 hybrid_clock.cc:648] HybridClock initialized: now 1790677718999759 us; error 30 us; skew 500 ppm
Sep 29 10:28:38 dist-test-slave-9cj5 krb5kdc[28315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677718, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:39.002764 28432 init.cc:377] Logged in from keytab as oryx/127.20.92.1@KRBTEST.COM (short username oryx)
I20260929 10:28:39.003755 28432 webserver.cc:460] Webserver started at http://127.20.92.1:38373/ using document root <none> and password file <none>
I20260929 10:28:39.004352 28432 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:39.004447 28432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:39.004683 28432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:39.006886 28432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-0/data/instance:
uuid: "a4679824ad5f48f4a46afdefd2c96a94"
format_stamp: "Formatted at 2026-09-29 10:28:38 on dist-test-slave-9cj5"
server_key: "445fe5e503628193dcf66446a8f9f5ce"
server_key_iv: "1801921b052594c3b6c2d370670457cf"
server_key_version: "encryptionkey@0"
I20260929 10:28:39.007468 28432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance:
uuid: "a4679824ad5f48f4a46afdefd2c96a94"
format_stamp: "Formatted at 2026-09-29 10:28:38 on dist-test-slave-9cj5"
server_key: "445fe5e503628193dcf66446a8f9f5ce"
server_key_iv: "1801921b052594c3b6c2d370670457cf"
server_key_version: "encryptionkey@0"
I20260929 10:28:39.011459 28432 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 10:28:39.014031 28447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:39.015372 28432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 10:28:39.015524 28432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "a4679824ad5f48f4a46afdefd2c96a94"
format_stamp: "Formatted at 2026-09-29 10:28:38 on dist-test-slave-9cj5"
server_key: "445fe5e503628193dcf66446a8f9f5ce"
server_key_iv: "1801921b052594c3b6c2d370670457cf"
server_key_version: "encryptionkey@0"
I20260929 10:28:39.015658 28432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:39.042802 28432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:39.046070 28432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:39.046286 28432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:39.047816 28432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:39.047897 28432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:39.047971 28432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:39.048026 28432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 10:28:39 dist-test-slave-9cj5 krb5kdc[28315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677718, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM
I20260929 10:28:39.059105 28432 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:46741
I20260929 10:28:39.059217 28562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:46741 every 8 connection(s)
I20260929 10:28:39.060034 28432 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
I20260929 10:28:39.062419 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:39.048738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:51709 (local address 127.20.92.62:39129)
0929 10:28:39.048876 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:39.048880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:39.050011 (+ 1131us) server_negotiation.cc:407] Connection header received
0929 10:28:39.051093 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:39.051097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:39.051164 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:39.051277 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:39.052975 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.053530 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.055063 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.055518 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.055646 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:39.058506 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:39.058509 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:39.058511 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:39.058536 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:39.060281 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:39.060926 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:39.060929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:39.060930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:39.061009 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:39.061342 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:39.061344 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:39.061345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:39.061516 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:39.061587 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:39.062166 (+ 579us) server_negotiation.cc:299] Negotiation successful
0929 10:28:39.062298 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 10:28:39.063179 28456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:39.049254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:39129 (local address 127.20.92.1:51709)
0929 10:28:39.049870 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:39.049913 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:39.050861 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:39.051513 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:39.051525 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:39.052016 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:39.052792 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:39.052816 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.053781 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.053786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:39.054902 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:39.054920 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.055704 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.055719 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:39.055967 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:39.056630 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:39.056646 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:39.058306 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:39.060396 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:39.060402 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:39.060419 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:39.060787 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:39.061101 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:39.061104 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:39.061106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:39.061251 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:39.061600 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:39.061605 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:39.061953 (+ 348us) client_negotiation.cc:769] Sending connection context
0929 10:28:39.062099 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 10:28:39.062343 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":123,"threads_started":1}
I20260929 10:28:39.064777 28563 heartbeater.cc:347] Connected to a master server at 127.20.92.62:39129
I20260929 10:28:39.064931 28563 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:39.065389 28563 heartbeater.cc:511] Master 127.20.92.62:39129 requested a full tablet report, sending...
I20260929 10:28:39.066334 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 28432
I20260929 10:28:39.066468 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance
I20260929 10:28:39.066718 20848 external_mini_cluster.cc:1428] Setting key 6e75cfcf2948abb9f6dc4e6c82d3dfe4
I20260929 10:28:39.067204 28359 ts_manager.cc:194] Registered new tserver with Master: a4679824ad5f48f4a46afdefd2c96a94 (127.20.92.1:46741)
I20260929 10:28:39.068742 28359 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.92.1@KRBTEST.COM'} at 127.20.92.1:51709
I20260929 10:28:39.075304 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:39.067229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:55521 (local address 127.20.92.62:39129)
0929 10:28:39.067375 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:39.067381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:39.068085 (+ 704us) server_negotiation.cc:407] Connection header received
0929 10:28:39.068297 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:39.068300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:39.068347 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:39.068438 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:39.068982 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.069870 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.070700 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.070931 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.071039 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:39.072184 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:39.072188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:39.072190 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:39.072222 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:39.073660 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:39.074101 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:39.074105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:39.074107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:39.074184 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:39.074467 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:39.074471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:39.074472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:39.074643 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:39.074744 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:39.075004 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 10:28:39.075114 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 10:28:39.075320 28569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:39.067260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:39129 (local address 127.20.92.1:55521)
0929 10:28:39.067997 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:39.068015 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:39.068157 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:39.068470 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:39.068473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:39.068726 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:39.068865 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:39.068875 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.070006 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.070009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:39.070581 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:39.070589 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.071044 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.071048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:39.071139 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:39.071694 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:39.071705 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:39.072037 (+ 332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:39.073794 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:39.073799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:39.073803 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:39.074001 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:39.074281 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:39.074286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:39.074288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:39.074372 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:39.074746 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:39.074749 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:39.074850 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 10:28:39.074955 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 10:28:39.075111 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for oryx/127.20.92.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.92.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.2.keytab.
Entry for principal oryx/127.20.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.2.keytab.
I20260929 10:28:39.095255 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:0
--local_ip_for_outbound_sockets=127.20.92.2
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:39129
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.2.keytab
--principal=oryx/127.20.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:36223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:39.209632 28572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:39.210000 28572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:39.210050 28572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:39.214200 28572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:39.214260 28572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:39.214305 28572 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 10:28:39.214340 28572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:28:39.219677 28572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:36223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.2.keytab
--principal=oryx/127.20.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.92.62:39129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:39.220757 28572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:39.221853 28572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:39.231911 28580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:39.232028 28578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:39.231911 28577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:39.232103 28572 server_base.cc:1034] running on GCE node
I20260929 10:28:39.232637 28572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:39.233352 28572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:39.234527 28572 hybrid_clock.cc:648] HybridClock initialized: now 1790677719234496 us; error 41 us; skew 500 ppm
Sep 29 10:28:39 dist-test-slave-9cj5 krb5kdc[28315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677719, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:39.237473 28572 init.cc:377] Logged in from keytab as oryx/127.20.92.2@KRBTEST.COM (short username oryx)
I20260929 10:28:39.238483 28572 webserver.cc:460] Webserver started at http://127.20.92.2:35989/ using document root <none> and password file <none>
I20260929 10:28:39.239079 28572 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:39.239171 28572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:39.239414 28572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:39.241680 28572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-1/data/instance:
uuid: "93dc1ff4066c4d21bd37e6241101305c"
format_stamp: "Formatted at 2026-09-29 10:28:39 on dist-test-slave-9cj5"
server_key: "69c087d40b98161acece8e605055fa21"
server_key_iv: "5c47d553958bec22263f450ead710af0"
server_key_version: "encryptionkey@0"
I20260929 10:28:39.242363 28572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance:
uuid: "93dc1ff4066c4d21bd37e6241101305c"
format_stamp: "Formatted at 2026-09-29 10:28:39 on dist-test-slave-9cj5"
server_key: "69c087d40b98161acece8e605055fa21"
server_key_iv: "5c47d553958bec22263f450ead710af0"
server_key_version: "encryptionkey@0"
I20260929 10:28:39.246182 28572 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 10:28:39.248682 28587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:39.249979 28572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:28:39.250115 28572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "93dc1ff4066c4d21bd37e6241101305c"
format_stamp: "Formatted at 2026-09-29 10:28:39 on dist-test-slave-9cj5"
server_key: "69c087d40b98161acece8e605055fa21"
server_key_iv: "5c47d553958bec22263f450ead710af0"
server_key_version: "encryptionkey@0"
I20260929 10:28:39.250232 28572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:39.260210 28572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:39.263305 28572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:39.263509 28572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:39.264901 28572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:39.264976 28572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:39.265045 28572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:39.265080 28572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:39.275914 28572 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:44115
Sep 29 10:28:39 dist-test-slave-9cj5 krb5kdc[28315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677719, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM
I20260929 10:28:39.275990 28702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:44115 every 8 connection(s)
I20260929 10:28:39.276870 28572 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
I20260929 10:28:39.280766 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:39.266067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:48049 (local address 127.20.92.62:39129)
0929 10:28:39.266220 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:39.266224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:39.267177 (+ 953us) server_negotiation.cc:407] Connection header received
0929 10:28:39.268364 (+ 1187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:39.268368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:39.268428 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:39.268516 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:39.270250 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.271211 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.273093 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.273323 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.273414 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:39.276672 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:39.276676 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:39.276679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:39.276711 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:39.278487 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:39.279051 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:39.279055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:39.279057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:39.279152 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:39.279538 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:39.279541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:39.279543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:39.279752 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:39.279880 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:39.280438 (+ 558us) server_negotiation.cc:299] Negotiation successful
0929 10:28:39.280576 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 10:28:39.281574 28596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:39.266517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:39129 (local address 127.20.92.2:48049)
0929 10:28:39.267005 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:39.267045 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:39.268135 (+ 1090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:39.268750 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:39.268761 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:39.269257 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:39.270038 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:39.270064 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.271357 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.271362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:39.272948 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:39.272959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.273489 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.273494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:39.273716 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:39.274543 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:39.274568 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:39.276478 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:39.278620 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:39.278627 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:39.278642 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:39.278927 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:39.279261 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:39.279265 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:39.279267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:39.279422 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:39.279877 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:39.279885 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:39.280166 (+ 281us) client_negotiation.cc:769] Sending connection context
0929 10:28:39.280377 (+ 211us) client_negotiation.cc:240] Negotiation successful
0929 10:28:39.280718 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":121,"threads_started":1}
I20260929 10:28:39.281988 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 28572
I20260929 10:28:39.282120 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance
I20260929 10:28:39.282444 20848 external_mini_cluster.cc:1428] Setting key 43eaadfe21b23c30e4e4a44a7a7fd00b
I20260929 10:28:39.283034 28703 heartbeater.cc:347] Connected to a master server at 127.20.92.62:39129
I20260929 10:28:39.283191 28703 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:39.283710 28703 heartbeater.cc:511] Master 127.20.92.62:39129 requested a full tablet report, sending...
I20260929 10:28:39.286458 28359 ts_manager.cc:194] Registered new tserver with Master: 93dc1ff4066c4d21bd37e6241101305c (127.20.92.2:44115)
I20260929 10:28:39.287091 28359 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.92.2@KRBTEST.COM'} at 127.20.92.2:48049
WARNING: no policy specified for oryx/127.20.92.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.92.3@KRBTEST.COM" created.
I20260929 10:28:39.298313 28710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:39.286164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:39129 (local address 127.20.92.2:44855)
0929 10:28:39.290044 (+ 3880us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:39.290063 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:39.290187 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:39.290491 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:39.290495 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:39.290779 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:39.290926 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:39.290936 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.291939 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.291943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:39.292854 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:39.292869 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.293387 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.293389 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:39.293465 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:39.294056 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:39.294068 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:39.294472 (+ 404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:39.296578 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:39.296581 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:39.296584 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:39.296794 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:39.297119 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:39.297123 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:39.297125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:39.297216 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:39.297673 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:39.297676 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:39.297805 (+ 129us) client_negotiation.cc:769] Sending connection context
0929 10:28:39.297946 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 10:28:39.298106 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3781,"thread_start_us":116,"threads_started":1}
I20260929 10:28:39.298300 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:39.286247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:44855 (local address 127.20.92.62:39129)
0929 10:28:39.286380 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:39.286385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:39.290157 (+ 3772us) server_negotiation.cc:407] Connection header received
0929 10:28:39.290279 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:39.290282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:39.290350 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:39.290425 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:39.291048 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.291760 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.293002 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.293279 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.293371 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:39.294648 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:39.294654 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:39.294663 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:39.294686 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:39.296468 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:39.296911 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:39.296916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:39.296918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:39.297011 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:39.297333 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:39.297336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:39.297338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:39.297566 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:39.297685 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:39.298005 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 10:28:39.298121 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.3.keytab.
Entry for principal oryx/127.20.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.3.keytab.
I20260929 10:28:39.313011 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:0
--local_ip_for_outbound_sockets=127.20.92.3
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:39129
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.3.keytab
--principal=oryx/127.20.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:36223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:39.425935 28712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:39.426221 28712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:39.426268 28712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:39.430697 28712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:39.430755 28712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:39.430795 28712 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 10:28:39.430828 28712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:28:39.436421 28712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:36223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.3.keytab
--principal=oryx/127.20.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.92.62:39129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:39.437479 28712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:39.438592 28712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:39.448524 28717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:39.448542 28718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:39.448794 28712 server_base.cc:1034] running on GCE node
W20260929 10:28:39.448529 28720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:39.449285 28712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:39.450003 28712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:39.451177 28712 hybrid_clock.cc:648] HybridClock initialized: now 1790677719451142 us; error 47 us; skew 500 ppm
Sep 29 10:28:39 dist-test-slave-9cj5 krb5kdc[28315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677719, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:39.454099 28712 init.cc:377] Logged in from keytab as oryx/127.20.92.3@KRBTEST.COM (short username oryx)
I20260929 10:28:39.455060 28712 webserver.cc:460] Webserver started at http://127.20.92.3:36047/ using document root <none> and password file <none>
I20260929 10:28:39.455631 28712 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:39.455729 28712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:39.455960 28712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:39.458069 28712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-2/data/instance:
uuid: "3a5bf278a026465bad9f17660385bff0"
format_stamp: "Formatted at 2026-09-29 10:28:39 on dist-test-slave-9cj5"
server_key: "1866875b85078e81018ff6c141f00213"
server_key_iv: "be33628355d6a06c487b1dc2fd12edf9"
server_key_version: "encryptionkey@0"
I20260929 10:28:39.458645 28712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance:
uuid: "3a5bf278a026465bad9f17660385bff0"
format_stamp: "Formatted at 2026-09-29 10:28:39 on dist-test-slave-9cj5"
server_key: "1866875b85078e81018ff6c141f00213"
server_key_iv: "be33628355d6a06c487b1dc2fd12edf9"
server_key_version: "encryptionkey@0"
I20260929 10:28:39.462622 28712 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 10:28:39.465201 28727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:39.466552 28712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:28:39.466691 28712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "3a5bf278a026465bad9f17660385bff0"
format_stamp: "Formatted at 2026-09-29 10:28:39 on dist-test-slave-9cj5"
server_key: "1866875b85078e81018ff6c141f00213"
server_key_iv: "be33628355d6a06c487b1dc2fd12edf9"
server_key_version: "encryptionkey@0"
I20260929 10:28:39.466809 28712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:39.482103 28712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:39.485334 28712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:39.485544 28712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:39.487037 28712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:39.487087 28712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:39.487126 28712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:39.487141 28712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:39.498055 28712 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:38451
I20260929 10:28:39.498126 28842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:38451 every 8 connection(s)
Sep 29 10:28:39 dist-test-slave-9cj5 krb5kdc[28315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677719, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM
I20260929 10:28:39.498960 28712 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:28:39.499725 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 28712
I20260929 10:28:39.500106 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance
I20260929 10:28:39.500424 20848 external_mini_cluster.cc:1428] Setting key 324cad71af2da4ab2ba5dceb6bda2839
I20260929 10:28:39.502944 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:39.488087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:46479 (local address 127.20.92.62:39129)
0929 10:28:39.488249 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:39.488255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:39.489142 (+ 887us) server_negotiation.cc:407] Connection header received
0929 10:28:39.490573 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:39.490580 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:39.490651 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:39.490761 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:39.492502 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.493396 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.495296 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.495514 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.495608 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:39.498827 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:39.498831 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:39.498834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:39.498866 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:39.500478 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:39.501086 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:39.501090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:39.501092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:39.501207 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:39.501648 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:39.501652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:39.501654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:39.501948 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:39.502036 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:39.502659 (+ 623us) server_negotiation.cc:299] Negotiation successful
0929 10:28:39.502810 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 10:28:39.503744 28736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:39.488506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:39129 (local address 127.20.92.3:46479)
0929 10:28:39.488984 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:39.489036 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:39.490309 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:39.490916 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:39.490923 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:39.491562 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:39.492312 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:39.492353 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.493552 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.493556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:39.495136 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:39.495148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.495715 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.495720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:39.495962 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:39.496743 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:39.496768 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:39.498637 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:39.500599 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:39.500605 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:39.500619 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:39.500984 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:39.501340 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:39.501346 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:39.501348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:39.501533 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:39.502059 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:39.502066 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:39.502366 (+ 300us) client_negotiation.cc:769] Sending connection context
0929 10:28:39.502571 (+ 205us) client_negotiation.cc:240] Negotiation successful
0929 10:28:39.502882 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":133,"threads_started":1}
I20260929 10:28:39.504877 28843 heartbeater.cc:347] Connected to a master server at 127.20.92.62:39129
I20260929 10:28:39.505024 28843 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:39.505602 28843 heartbeater.cc:511] Master 127.20.92.62:39129 requested a full tablet report, sending...
I20260929 10:28:39.506884 28360 ts_manager.cc:194] Registered new tserver with Master: 3a5bf278a026465bad9f17660385bff0 (127.20.92.3:38451)
I20260929 10:28:39.507754 28360 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.92.3@KRBTEST.COM'} at 127.20.92.3:46479
I20260929 10:28:39.514739 20848 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:28:39.515857 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:39.507452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:54081 (local address 127.20.92.62:39129)
0929 10:28:39.507586 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:39.507592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:39.508208 (+ 616us) server_negotiation.cc:407] Connection header received
0929 10:28:39.508357 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:39.508365 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:39.508406 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:39.508629 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:39.509087 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.509587 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.510544 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.510756 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.510885 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:39.512009 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:39.512013 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:39.512016 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:39.512044 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:39.513860 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:39.514389 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:39.514392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:39.514394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:39.514482 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:39.514788 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:39.514793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:39.514795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:39.515015 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:39.515135 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:39.515478 (+ 343us) server_negotiation.cc:299] Negotiation successful
0929 10:28:39.515610 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 10:28:39.515882 28849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:39.507312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:39129 (local address 127.20.92.3:54081)
0929 10:28:39.508098 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:39.508123 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:39.508253 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:39.508639 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:39.508643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:39.508859 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:39.508966 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:39.508972 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.509712 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.509723 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:39.510422 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:39.510432 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.510887 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.510891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:39.510989 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:39.511495 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:39.511506 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:39.511852 (+ 346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:39.513974 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:39.513980 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:39.513983 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:39.514268 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:39.514574 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:39.514577 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:39.514581 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:39.514675 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:39.515143 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:39.515148 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:39.515251 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 10:28:39.515399 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 10:28:39.515576 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":2,"thread_start_us":119,"threads_started":1}
Sep 29 10:28:39 dist-test-slave-9cj5 krb5kdc[28315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677719, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 10:28:39 dist-test-slave-9cj5 krb5kdc[28315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:39 dist-test-slave-9cj5 krb5kdc[28315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 10:28:39.539659 28409 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:39.533278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41998 (local address 127.20.92.62:39129)
0929 10:28:39.533422 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:39.533426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:39.533439 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:28:39.533481 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:39.533484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:39.533522 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:39.533595 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:39.534412 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.534994 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.535881 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.536101 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.536196 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:39.539497 (+ 3301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41998: BlockingRecv error: recv got EOF from 127.0.0.1:41998 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 10:28:39 dist-test-slave-9cj5 krb5kdc[28315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677719, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM
I20260929 10:28:39.549469 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:39.541117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42006 (local address 127.20.92.62:39129)
0929 10:28:39.541255 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:39.541261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:39.541406 (+ 145us) server_negotiation.cc:407] Connection header received
0929 10:28:39.541559 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:39.541562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:39.541611 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:39.541700 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:39.542389 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.542920 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.543729 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.543950 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.544061 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:39.546425 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:39.546435 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:39.546438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:39.546462 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:39.547746 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:39.548202 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:39.548206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:39.548207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:39.548290 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:39.548593 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:39.548596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:39.548598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:39.548782 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:39.548930 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:39.549166 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 10:28:39.549281 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 10:28:39.552367 28360 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:28:39.555608 28360 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:28:39.569919 28867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:39.564025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:44115 (local address 127.0.0.1:51392)
0929 10:28:39.564669 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:39.564683 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:39.564783 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:39.565933 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:39.565935 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:39.565951 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:39.566085 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:39.566090 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.567094 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.567100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:39.568440 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:39.568451 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.569347 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.569350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:39.569453 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:39.569490 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 10:28:39.569610 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 10:28:39.569733 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":97,"threads_started":1}
I20260929 10:28:39.569929 28869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:39.564666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51392 (local address 127.20.92.2:44115)
0929 10:28:39.565385 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:39.565399 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:39.565422 (+ 23us) server_negotiation.cc:407] Connection header received
0929 10:28:39.565495 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:39.565500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:39.565696 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:39.566186 (+ 490us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:39.566287 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.566976 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.568579 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.569219 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.569345 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:39.569515 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:39.569651 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 10:28:39.569739 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":134,"threads_started":1}
I20260929 10:28:39.570415 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:39.563875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:46741 (local address 127.0.0.1:40434)
0929 10:28:39.564390 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:39.564439 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:39.564689 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:39.565605 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:39.565612 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:39.565634 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:39.565824 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:39.565832 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.567532 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.567535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:39.568704 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:39.568713 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.569907 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.569915 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:39.570029 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:39.570057 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 10:28:39.570109 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 10:28:39.570273 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":131,"thread_start_us":74,"threads_started":1}
I20260929 10:28:39.570832 28866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:39.563948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40434 (local address 127.20.92.1:46741)
0929 10:28:39.565083 (+ 1135us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:39.565090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:39.565120 (+ 30us) server_negotiation.cc:407] Connection header received
0929 10:28:39.565208 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:39.565215 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:39.565438 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:39.566006 (+ 568us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:39.566142 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.567300 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.568807 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.569718 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.570265 (+ 547us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:39.570418 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:39.570479 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 10:28:39.570575 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1031,"thread_start_us":79,"threads_started":1}
I20260929 10:28:39.570860 28870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:39.565030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44946 (local address 127.20.92.3:38451)
0929 10:28:39.565355 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:39.565366 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:39.565385 (+ 19us) server_negotiation.cc:407] Connection header received
0929 10:28:39.565451 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:39.565455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:39.565627 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:39.566081 (+ 454us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:39.566202 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.567323 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.568986 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.569870 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.570292 (+ 422us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:39.570485 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:39.570533 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 10:28:39.570614 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":117,"threads_started":1}
I20260929 10:28:39.570878 28868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:39.564565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:38451 (local address 127.0.0.1:44946)
0929 10:28:39.565011 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:39.565024 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:39.565104 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:39.565797 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:39.565801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:39.565815 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:39.565929 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:39.565935 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.567576 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.567579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:39.568843 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:39.568853 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.569995 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.569998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:39.570077 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:39.570110 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 10:28:39.570154 (+ 44us) client_negotiation.cc:240] Negotiation successful
0929 10:28:39.570706 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":445,"thread_start_us":90,"threads_started":1}
I20260929 10:28:39.574334 28637 tablet_service.cc:1467] Processing CreateTablet for tablet 17f757a9917646de9b47a83027785a52 (DEFAULT_TABLE table=test-table [id=c389143d19cb40668c60f81afe7b7cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:39.574252 28777 tablet_service.cc:1467] Processing CreateTablet for tablet 17f757a9917646de9b47a83027785a52 (DEFAULT_TABLE table=test-table [id=c389143d19cb40668c60f81afe7b7cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:39.574237 28497 tablet_service.cc:1467] Processing CreateTablet for tablet 17f757a9917646de9b47a83027785a52 (DEFAULT_TABLE table=test-table [id=c389143d19cb40668c60f81afe7b7cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:39.575277 28637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17f757a9917646de9b47a83027785a52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:39.575349 28777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17f757a9917646de9b47a83027785a52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:39.575343 28497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17f757a9917646de9b47a83027785a52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:39.582451 28871 tablet_bootstrap.cc:492] T 17f757a9917646de9b47a83027785a52 P 93dc1ff4066c4d21bd37e6241101305c: Bootstrap starting.
I20260929 10:28:39.583938 28872 tablet_bootstrap.cc:492] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94: Bootstrap starting.
I20260929 10:28:39.585021 28871 tablet_bootstrap.cc:654] T 17f757a9917646de9b47a83027785a52 P 93dc1ff4066c4d21bd37e6241101305c: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:39.585783 28871 log.cc:824] T 17f757a9917646de9b47a83027785a52 P 93dc1ff4066c4d21bd37e6241101305c: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:39.585966 28872 tablet_bootstrap.cc:654] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:39.586094 28873 tablet_bootstrap.cc:492] T 17f757a9917646de9b47a83027785a52 P 3a5bf278a026465bad9f17660385bff0: Bootstrap starting.
I20260929 10:28:39.586826 28872 log.cc:824] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:39.588044 28873 tablet_bootstrap.cc:654] T 17f757a9917646de9b47a83027785a52 P 3a5bf278a026465bad9f17660385bff0: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:39.588411 28871 tablet_bootstrap.cc:492] T 17f757a9917646de9b47a83027785a52 P 93dc1ff4066c4d21bd37e6241101305c: No bootstrap required, opened a new log
I20260929 10:28:39.588598 28871 ts_tablet_manager.cc:1397] T 17f757a9917646de9b47a83027785a52 P 93dc1ff4066c4d21bd37e6241101305c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 10:28:39.588781 28872 tablet_bootstrap.cc:492] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94: No bootstrap required, opened a new log
I20260929 10:28:39.588824 28873 log.cc:824] T 17f757a9917646de9b47a83027785a52 P 3a5bf278a026465bad9f17660385bff0: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:39.588985 28872 ts_tablet_manager.cc:1397] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 10:28:39.590883 28873 tablet_bootstrap.cc:492] T 17f757a9917646de9b47a83027785a52 P 3a5bf278a026465bad9f17660385bff0: No bootstrap required, opened a new log
I20260929 10:28:39.591060 28873 ts_tablet_manager.cc:1397] T 17f757a9917646de9b47a83027785a52 P 3a5bf278a026465bad9f17660385bff0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 10:28:39.592685 28871 raft_consensus.cc:348] T 17f757a9917646de9b47a83027785a52 P 93dc1ff4066c4d21bd37e6241101305c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:39.592970 28871 raft_consensus.cc:374] T 17f757a9917646de9b47a83027785a52 P 93dc1ff4066c4d21bd37e6241101305c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:39.593056 28871 raft_consensus.cc:729] T 17f757a9917646de9b47a83027785a52 P 93dc1ff4066c4d21bd37e6241101305c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93dc1ff4066c4d21bd37e6241101305c, State: Initialized, Role: FOLLOWER
I20260929 10:28:39.593338 28872 raft_consensus.cc:348] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:39.593616 28872 raft_consensus.cc:374] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:39.593552 28871 consensus_queue.cc:260] T 17f757a9917646de9b47a83027785a52 P 93dc1ff4066c4d21bd37e6241101305c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:39.593710 28872 raft_consensus.cc:729] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4679824ad5f48f4a46afdefd2c96a94, State: Initialized, Role: FOLLOWER
I20260929 10:28:39.594206 28872 consensus_queue.cc:260] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:39.594928 28703 heartbeater.cc:503] Master 127.20.92.62:39129 was elected leader, sending a full tablet report...
I20260929 10:28:39.594884 28873 raft_consensus.cc:348] T 17f757a9917646de9b47a83027785a52 P 3a5bf278a026465bad9f17660385bff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:39.595161 28873 raft_consensus.cc:374] T 17f757a9917646de9b47a83027785a52 P 3a5bf278a026465bad9f17660385bff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:39.595211 28871 ts_tablet_manager.cc:1428] T 17f757a9917646de9b47a83027785a52 P 93dc1ff4066c4d21bd37e6241101305c: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 10:28:39.595244 28873 raft_consensus.cc:729] T 17f757a9917646de9b47a83027785a52 P 3a5bf278a026465bad9f17660385bff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5bf278a026465bad9f17660385bff0, State: Initialized, Role: FOLLOWER
I20260929 10:28:39.595409 28872 ts_tablet_manager.cc:1428] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 10:28:39.595700 28873 consensus_queue.cc:260] T 17f757a9917646de9b47a83027785a52 P 3a5bf278a026465bad9f17660385bff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:39.595845 28563 heartbeater.cc:503] Master 127.20.92.62:39129 was elected leader, sending a full tablet report...
I20260929 10:28:39.596369 28843 heartbeater.cc:503] Master 127.20.92.62:39129 was elected leader, sending a full tablet report...
I20260929 10:28:39.596582 28873 ts_tablet_manager.cc:1428] T 17f757a9917646de9b47a83027785a52 P 3a5bf278a026465bad9f17660385bff0: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260929 10:28:39.728971 28878 raft_consensus.cc:482] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:28:39.729194 28878 raft_consensus.cc:504] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:39.730356 28878 leader_election.cc:288] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93dc1ff4066c4d21bd37e6241101305c (127.20.92.2:44115), 3a5bf278a026465bad9f17660385bff0 (127.20.92.3:38451)
I20260929 10:28:39.734383 28869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:39.730737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:56419 (local address 127.20.92.2:44115)
0929 10:28:39.730878 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:39.730884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:39.731132 (+ 248us) server_negotiation.cc:407] Connection header received
0929 10:28:39.731305 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:39.731308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:39.731352 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:39.731437 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:39.731749 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.732284 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.733365 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.733821 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.733942 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:39.734006 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:39.734193 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 10:28:39.734259 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 10:28:39.734424 28880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:39.730612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:44115 (local address 127.20.92.1:56419)
0929 10:28:39.731028 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:39.731040 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:39.731180 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:39.731461 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:39.731464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:39.731479 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:39.731634 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:39.731640 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.732407 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.732412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:39.733231 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:39.733240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.733945 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.733949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:39.734048 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:39.734079 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 10:28:39.734166 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 10:28:39.734241 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":24,"spinlock_wait_cycles":6528,"thread_start_us":149,"threads_started":1}
I20260929 10:28:39.734623 28870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:39.730752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:51539 (local address 127.20.92.3:38451)
0929 10:28:39.730902 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:39.730907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:39.731154 (+ 247us) server_negotiation.cc:407] Connection header received
0929 10:28:39.731309 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:39.731313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:39.731368 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:39.731456 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:39.731766 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.732285 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.733581 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.734082 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.734205 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:39.734263 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:39.734446 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 10:28:39.734518 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 10:28:39.734642 28881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:39.730625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:38451 (local address 127.20.92.1:51539)
0929 10:28:39.731052 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:39.731063 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:39.731195 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:39.731468 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:39.731470 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:39.731486 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:39.731649 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:39.731656 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.732631 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.732637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:39.733450 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:39.733459 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.734209 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.734213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:39.734308 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:39.734341 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 10:28:39.734431 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 10:28:39.734496 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":17,"thread_start_us":135,"threads_started":1}
I20260929 10:28:39.735185 28657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17f757a9917646de9b47a83027785a52" candidate_uuid: "a4679824ad5f48f4a46afdefd2c96a94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93dc1ff4066c4d21bd37e6241101305c" is_pre_election: true
I20260929 10:28:39.735275 28796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17f757a9917646de9b47a83027785a52" candidate_uuid: "a4679824ad5f48f4a46afdefd2c96a94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5bf278a026465bad9f17660385bff0" is_pre_election: true
I20260929 10:28:39.735534 28657 raft_consensus.cc:2454] T 17f757a9917646de9b47a83027785a52 P 93dc1ff4066c4d21bd37e6241101305c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4679824ad5f48f4a46afdefd2c96a94 in term 0.
I20260929 10:28:39.735553 28796 raft_consensus.cc:2454] T 17f757a9917646de9b47a83027785a52 P 3a5bf278a026465bad9f17660385bff0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4679824ad5f48f4a46afdefd2c96a94 in term 0.
I20260929 10:28:39.736069 28448 leader_election.cc:302] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3a5bf278a026465bad9f17660385bff0, 93dc1ff4066c4d21bd37e6241101305c, a4679824ad5f48f4a46afdefd2c96a94; no voters:
I20260929 10:28:39.736370 28878 raft_consensus.cc:2781] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:28:39.736451 28878 raft_consensus.cc:482] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:28:39.736485 28878 raft_consensus.cc:3037] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:39.737527 28878 raft_consensus.cc:504] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:39.737998 28878 leader_election.cc:288] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94 [CANDIDATE]: Term 1 election: Requested vote from peers 93dc1ff4066c4d21bd37e6241101305c (127.20.92.2:44115), 3a5bf278a026465bad9f17660385bff0 (127.20.92.3:38451)
I20260929 10:28:39.738345 28657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17f757a9917646de9b47a83027785a52" candidate_uuid: "a4679824ad5f48f4a46afdefd2c96a94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93dc1ff4066c4d21bd37e6241101305c"
I20260929 10:28:39.738492 28796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17f757a9917646de9b47a83027785a52" candidate_uuid: "a4679824ad5f48f4a46afdefd2c96a94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5bf278a026465bad9f17660385bff0"
I20260929 10:28:39.738497 28657 raft_consensus.cc:3037] T 17f757a9917646de9b47a83027785a52 P 93dc1ff4066c4d21bd37e6241101305c [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:39.738616 28796 raft_consensus.cc:3037] T 17f757a9917646de9b47a83027785a52 P 3a5bf278a026465bad9f17660385bff0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:39.739648 28796 raft_consensus.cc:2454] T 17f757a9917646de9b47a83027785a52 P 3a5bf278a026465bad9f17660385bff0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4679824ad5f48f4a46afdefd2c96a94 in term 1.
I20260929 10:28:39.740021 28450 leader_election.cc:302] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a5bf278a026465bad9f17660385bff0, a4679824ad5f48f4a46afdefd2c96a94; no voters:
I20260929 10:28:39.740119 28657 raft_consensus.cc:2454] T 17f757a9917646de9b47a83027785a52 P 93dc1ff4066c4d21bd37e6241101305c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4679824ad5f48f4a46afdefd2c96a94 in term 1.
I20260929 10:28:39.740218 28878 raft_consensus.cc:2781] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:39.740443 28878 raft_consensus.cc:686] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94 [term 1 LEADER]: Becoming Leader. State: Replica: a4679824ad5f48f4a46afdefd2c96a94, State: Running, Role: LEADER
I20260929 10:28:39.740814 28878 consensus_queue.cc:237] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:39.744383 28359 catalog_manager.cc:5495] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4679824ad5f48f4a46afdefd2c96a94 (127.20.92.1). New cstate: current_term: 1 leader_uuid: "a4679824ad5f48f4a46afdefd2c96a94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } health_report { overall_health: UNKNOWN } } }
W20260929 10:28:39.751556 28844 tablet.cc:2369] T 17f757a9917646de9b47a83027785a52 P 3a5bf278a026465bad9f17660385bff0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:28:39.761709 28866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:39.758046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40444 (local address 127.20.92.1:46741)
0929 10:28:39.758181 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:39.758185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:39.758199 (+ 14us) server_negotiation.cc:407] Connection header received
0929 10:28:39.758281 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:39.758283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:39.758339 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:39.758440 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:28:39.758727 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.759263 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.760150 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.760375 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.760461 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:39.760754 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:28:39.761234 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:28:39.761344 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:39.761455 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 10:28:39.761511 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 10:28:39.768829 28796 raft_consensus.cc:1261] T 17f757a9917646de9b47a83027785a52 P 3a5bf278a026465bad9f17660385bff0 [term 1 FOLLOWER]: Refusing update from remote peer a4679824ad5f48f4a46afdefd2c96a94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:39.768859 28657 raft_consensus.cc:1261] T 17f757a9917646de9b47a83027785a52 P 93dc1ff4066c4d21bd37e6241101305c [term 1 FOLLOWER]: Refusing update from remote peer a4679824ad5f48f4a46afdefd2c96a94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:39.769665 28882 consensus_queue.cc:1035] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:28:39.770004 28878 consensus_queue.cc:1035] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 10:28:39.778015 28704 tablet.cc:2369] T 17f757a9917646de9b47a83027785a52 P 93dc1ff4066c4d21bd37e6241101305c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:28:39.780521 28885 mvcc.cc:205] Tried to move back new op lower bound from 7334615940164907008 to 7334615940061241344. Current Snapshot: MvccSnapshot[applied={T|T < 7334615940164907008}]
I20260929 10:28:39.780985 28887 mvcc.cc:205] Tried to move back new op lower bound from 7334615940164907008 to 7334615940061241344. Current Snapshot: MvccSnapshot[applied={T|T < 7334615940164907008}]
I20260929 10:28:39.784948 28886 mvcc.cc:205] Tried to move back new op lower bound from 7334615940164907008 to 7334615940061241344. Current Snapshot: MvccSnapshot[applied={T|T < 7334615940164907008}]
I20260929 10:28:39.790870 28359 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:28:39.791136 28359 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:28:39.793898 28359 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4bcb22f0ca84d1e92a09724ab98d198: Sending DeleteTablet for 3 replicas of tablet 17f757a9917646de9b47a83027785a52
I20260929 10:28:39.794729 28497 tablet_service.cc:1514] Processing DeleteTablet for tablet 17f757a9917646de9b47a83027785a52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:39 UTC) from {username='oryx', principal='oryx/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:40434
I20260929 10:28:39.794893 28637 tablet_service.cc:1514] Processing DeleteTablet for tablet 17f757a9917646de9b47a83027785a52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:39 UTC) from {username='oryx', principal='oryx/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:51392
I20260929 10:28:39.794917 28777 tablet_service.cc:1514] Processing DeleteTablet for tablet 17f757a9917646de9b47a83027785a52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:39 UTC) from {username='oryx', principal='oryx/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:44946
I20260929 10:28:39.795413 28894 tablet_replica.cc:330] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94: stopping tablet replica
I20260929 10:28:39.795472 28895 tablet_replica.cc:330] T 17f757a9917646de9b47a83027785a52 P 93dc1ff4066c4d21bd37e6241101305c: stopping tablet replica
I20260929 10:28:39.795778 28894 raft_consensus.cc:2229] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:28:39.795778 28895 raft_consensus.cc:2229] T 17f757a9917646de9b47a83027785a52 P 93dc1ff4066c4d21bd37e6241101305c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:28:39.796103 28895 raft_consensus.cc:2258] T 17f757a9917646de9b47a83027785a52 P 93dc1ff4066c4d21bd37e6241101305c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:39.796062 28359 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 10:28:39.796494 28896 tablet_replica.cc:330] T 17f757a9917646de9b47a83027785a52 P 3a5bf278a026465bad9f17660385bff0: stopping tablet replica
W20260929 10:28:39.796492 28359 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:28:39.796775 28896 raft_consensus.cc:2229] T 17f757a9917646de9b47a83027785a52 P 3a5bf278a026465bad9f17660385bff0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:28:39.797065 28896 raft_consensus.cc:2258] T 17f757a9917646de9b47a83027785a52 P 3a5bf278a026465bad9f17660385bff0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:39.796288 28894 raft_consensus.cc:2258] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:39.798790 28894 ts_tablet_manager.cc:1905] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:39.798961 28896 ts_tablet_manager.cc:1905] T 17f757a9917646de9b47a83027785a52 P 3a5bf278a026465bad9f17660385bff0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:39.801656 28896 ts_tablet_manager.cc:1918] T 17f757a9917646de9b47a83027785a52 P 3a5bf278a026465bad9f17660385bff0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:28:39.801837 28896 log.cc:1196] T 17f757a9917646de9b47a83027785a52 P 3a5bf278a026465bad9f17660385bff0: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-2/wal/wals/17f757a9917646de9b47a83027785a52
I20260929 10:28:39.802220 28896 ts_tablet_manager.cc:1939] T 17f757a9917646de9b47a83027785a52 P 3a5bf278a026465bad9f17660385bff0: Deleting consensus metadata
I20260929 10:28:39.802841 28894 ts_tablet_manager.cc:1918] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:28:39.803068 28894 log.cc:1196] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-0/wal/wals/17f757a9917646de9b47a83027785a52
I20260929 10:28:39.803709 28894 ts_tablet_manager.cc:1939] T 17f757a9917646de9b47a83027785a52 P a4679824ad5f48f4a46afdefd2c96a94: Deleting consensus metadata
I20260929 10:28:39.805127 28346 catalog_manager.cc:4841] TS 3a5bf278a026465bad9f17660385bff0 (127.20.92.3:38451): tablet 17f757a9917646de9b47a83027785a52 (table test-table [id=c389143d19cb40668c60f81afe7b7cc7]) successfully deleted
I20260929 10:28:39.806360 28895 ts_tablet_manager.cc:1905] T 17f757a9917646de9b47a83027785a52 P 93dc1ff4066c4d21bd37e6241101305c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:39.807027 28345 catalog_manager.cc:4841] TS a4679824ad5f48f4a46afdefd2c96a94 (127.20.92.1:46741): tablet 17f757a9917646de9b47a83027785a52 (table test-table [id=c389143d19cb40668c60f81afe7b7cc7]) successfully deleted
I20260929 10:28:39.809290 28895 ts_tablet_manager.cc:1918] T 17f757a9917646de9b47a83027785a52 P 93dc1ff4066c4d21bd37e6241101305c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:28:39.809427 28895 log.cc:1196] T 17f757a9917646de9b47a83027785a52 P 93dc1ff4066c4d21bd37e6241101305c: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-1/wal/wals/17f757a9917646de9b47a83027785a52
I20260929 10:28:39.809450 28497 tablet_service.cc:1467] Processing CreateTablet for tablet 0c1bd32ddcff42e28f83d1685a4a6e0d (DEFAULT_TABLE table=test-table [id=634d3ff3146642b8a4e095b1d7d29b19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:39.809530 28637 tablet_service.cc:1467] Processing CreateTablet for tablet 0c1bd32ddcff42e28f83d1685a4a6e0d (DEFAULT_TABLE table=test-table [id=634d3ff3146642b8a4e095b1d7d29b19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:39.809805 28497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c1bd32ddcff42e28f83d1685a4a6e0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:39.809828 28895 ts_tablet_manager.cc:1939] T 17f757a9917646de9b47a83027785a52 P 93dc1ff4066c4d21bd37e6241101305c: Deleting consensus metadata
I20260929 10:28:39.809904 28637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c1bd32ddcff42e28f83d1685a4a6e0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:39.811607 28777 tablet_service.cc:1467] Processing CreateTablet for tablet 0c1bd32ddcff42e28f83d1685a4a6e0d (DEFAULT_TABLE table=test-table [id=634d3ff3146642b8a4e095b1d7d29b19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:39.811895 28777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c1bd32ddcff42e28f83d1685a4a6e0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:39.812288 28872 tablet_bootstrap.cc:492] T 0c1bd32ddcff42e28f83d1685a4a6e0d P a4679824ad5f48f4a46afdefd2c96a94: Bootstrap starting.
I20260929 10:28:39.812312 28871 tablet_bootstrap.cc:492] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 93dc1ff4066c4d21bd37e6241101305c: Bootstrap starting.
I20260929 10:28:39.813196 28871 tablet_bootstrap.cc:654] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 93dc1ff4066c4d21bd37e6241101305c: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:39.813272 28872 tablet_bootstrap.cc:654] T 0c1bd32ddcff42e28f83d1685a4a6e0d P a4679824ad5f48f4a46afdefd2c96a94: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:39.813962 28344 catalog_manager.cc:4841] TS 93dc1ff4066c4d21bd37e6241101305c (127.20.92.2:44115): tablet 17f757a9917646de9b47a83027785a52 (table test-table [id=c389143d19cb40668c60f81afe7b7cc7]) successfully deleted
I20260929 10:28:39.814365 28873 tablet_bootstrap.cc:492] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 3a5bf278a026465bad9f17660385bff0: Bootstrap starting.
I20260929 10:28:39.814706 28872 tablet_bootstrap.cc:492] T 0c1bd32ddcff42e28f83d1685a4a6e0d P a4679824ad5f48f4a46afdefd2c96a94: No bootstrap required, opened a new log
I20260929 10:28:39.814870 28872 ts_tablet_manager.cc:1397] T 0c1bd32ddcff42e28f83d1685a4a6e0d P a4679824ad5f48f4a46afdefd2c96a94: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 10:28:39.815280 28873 tablet_bootstrap.cc:654] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 3a5bf278a026465bad9f17660385bff0: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:39.815402 28872 raft_consensus.cc:348] T 0c1bd32ddcff42e28f83d1685a4a6e0d P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:39.815526 28872 raft_consensus.cc:374] T 0c1bd32ddcff42e28f83d1685a4a6e0d P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:39.815565 28872 raft_consensus.cc:729] T 0c1bd32ddcff42e28f83d1685a4a6e0d P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4679824ad5f48f4a46afdefd2c96a94, State: Initialized, Role: FOLLOWER
I20260929 10:28:39.815699 28872 consensus_queue.cc:260] T 0c1bd32ddcff42e28f83d1685a4a6e0d P a4679824ad5f48f4a46afdefd2c96a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:39.816025 28872 ts_tablet_manager.cc:1428] T 0c1bd32ddcff42e28f83d1685a4a6e0d P a4679824ad5f48f4a46afdefd2c96a94: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 10:28:39.816283 28871 tablet_bootstrap.cc:492] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 93dc1ff4066c4d21bd37e6241101305c: No bootstrap required, opened a new log
I20260929 10:28:39.816355 28871 ts_tablet_manager.cc:1397] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 93dc1ff4066c4d21bd37e6241101305c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 10:28:39.816668 28873 tablet_bootstrap.cc:492] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 3a5bf278a026465bad9f17660385bff0: No bootstrap required, opened a new log
I20260929 10:28:39.816742 28873 ts_tablet_manager.cc:1397] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 3a5bf278a026465bad9f17660385bff0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260929 10:28:39.816756 28871 raft_consensus.cc:348] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 93dc1ff4066c4d21bd37e6241101305c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:39.816865 28871 raft_consensus.cc:374] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 93dc1ff4066c4d21bd37e6241101305c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:39.816903 28871 raft_consensus.cc:729] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 93dc1ff4066c4d21bd37e6241101305c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93dc1ff4066c4d21bd37e6241101305c, State: Initialized, Role: FOLLOWER
I20260929 10:28:39.817022 28871 consensus_queue.cc:260] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 93dc1ff4066c4d21bd37e6241101305c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:39.817217 28873 raft_consensus.cc:348] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 3a5bf278a026465bad9f17660385bff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:39.817292 28871 ts_tablet_manager.cc:1428] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 93dc1ff4066c4d21bd37e6241101305c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:28:39.817322 28873 raft_consensus.cc:374] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 3a5bf278a026465bad9f17660385bff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:39.817356 28873 raft_consensus.cc:729] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 3a5bf278a026465bad9f17660385bff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5bf278a026465bad9f17660385bff0, State: Initialized, Role: FOLLOWER
I20260929 10:28:39.817485 28873 consensus_queue.cc:260] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 3a5bf278a026465bad9f17660385bff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:39.817726 28873 ts_tablet_manager.cc:1428] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 3a5bf278a026465bad9f17660385bff0: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260929 10:28:39.925088 28879 raft_consensus.cc:482] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 3a5bf278a026465bad9f17660385bff0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:28:39.925292 28879 raft_consensus.cc:504] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 3a5bf278a026465bad9f17660385bff0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:39.926499 28879 leader_election.cc:288] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 3a5bf278a026465bad9f17660385bff0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4679824ad5f48f4a46afdefd2c96a94 (127.20.92.1:46741), 93dc1ff4066c4d21bd37e6241101305c (127.20.92.2:44115)
I20260929 10:28:39.930994 28869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:39.927190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:56293 (local address 127.20.92.2:44115)
0929 10:28:39.927315 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:39.927320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:39.927336 (+ 16us) server_negotiation.cc:407] Connection header received
0929 10:28:39.927410 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:39.927414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:39.927463 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:39.927554 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:39.927846 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.928572 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.930007 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.930397 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.930527 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:39.930592 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:39.930792 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 10:28:39.930863 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 10:28:39.931010 28900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:39.926805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:44115 (local address 127.20.92.3:56293)
0929 10:28:39.927087 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:39.927099 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:39.927192 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:39.927579 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:39.927582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:39.927595 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:39.927717 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:39.927722 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.928700 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.928703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:39.929889 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:39.929896 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.930539 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.930542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:39.930642 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:39.930676 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 10:28:39.930776 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 10:28:39.930857 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":105,"threads_started":1}
I20260929 10:28:39.931247 28866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:39.926793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:51681 (local address 127.20.92.1:46741)
0929 10:28:39.926911 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:39.926917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:39.927122 (+ 205us) server_negotiation.cc:407] Connection header received
0929 10:28:39.927188 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:39.927191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:39.927245 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:39.927311 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:39.927723 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.928253 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.929846 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.930442 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.930543 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:39.930604 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:39.931060 (+ 456us) server_negotiation.cc:299] Negotiation successful
0929 10:28:39.931134 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 10:28:39.931488 28736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:39.926722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:46741 (local address 127.20.92.3:51681)
0929 10:28:39.926909 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:39.926925 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:39.927028 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:39.927434 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:39.927437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:39.927451 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:39.927578 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:39.927585 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.928372 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.928377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:39.929699 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:39.929709 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.930663 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.930666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:39.930795 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:39.930816 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 10:28:39.930881 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 10:28:39.930948 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260929 10:28:39.931737 28656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c1bd32ddcff42e28f83d1685a4a6e0d" candidate_uuid: "3a5bf278a026465bad9f17660385bff0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93dc1ff4066c4d21bd37e6241101305c" is_pre_election: true
I20260929 10:28:39.931907 28656 raft_consensus.cc:2454] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 93dc1ff4066c4d21bd37e6241101305c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a5bf278a026465bad9f17660385bff0 in term 0.
I20260929 10:28:39.931993 28517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c1bd32ddcff42e28f83d1685a4a6e0d" candidate_uuid: "3a5bf278a026465bad9f17660385bff0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4679824ad5f48f4a46afdefd2c96a94" is_pre_election: true
I20260929 10:28:39.932207 28517 raft_consensus.cc:2454] T 0c1bd32ddcff42e28f83d1685a4a6e0d P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a5bf278a026465bad9f17660385bff0 in term 0.
I20260929 10:28:39.932310 28728 leader_election.cc:302] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 3a5bf278a026465bad9f17660385bff0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a5bf278a026465bad9f17660385bff0, 93dc1ff4066c4d21bd37e6241101305c; no voters:
I20260929 10:28:39.932617 28879 raft_consensus.cc:2781] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 3a5bf278a026465bad9f17660385bff0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:28:39.932679 28879 raft_consensus.cc:482] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 3a5bf278a026465bad9f17660385bff0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:28:39.932713 28879 raft_consensus.cc:3037] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 3a5bf278a026465bad9f17660385bff0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:39.934001 28879 raft_consensus.cc:504] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 3a5bf278a026465bad9f17660385bff0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:39.934363 28879 leader_election.cc:288] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 3a5bf278a026465bad9f17660385bff0 [CANDIDATE]: Term 1 election: Requested vote from peers a4679824ad5f48f4a46afdefd2c96a94 (127.20.92.1:46741), 93dc1ff4066c4d21bd37e6241101305c (127.20.92.2:44115)
I20260929 10:28:39.934692 28517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c1bd32ddcff42e28f83d1685a4a6e0d" candidate_uuid: "3a5bf278a026465bad9f17660385bff0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4679824ad5f48f4a46afdefd2c96a94"
I20260929 10:28:39.934726 28656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c1bd32ddcff42e28f83d1685a4a6e0d" candidate_uuid: "3a5bf278a026465bad9f17660385bff0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93dc1ff4066c4d21bd37e6241101305c"
I20260929 10:28:39.934834 28656 raft_consensus.cc:3037] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 93dc1ff4066c4d21bd37e6241101305c [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:39.934842 28517 raft_consensus.cc:3037] T 0c1bd32ddcff42e28f83d1685a4a6e0d P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:39.936002 28656 raft_consensus.cc:2454] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 93dc1ff4066c4d21bd37e6241101305c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a5bf278a026465bad9f17660385bff0 in term 1.
I20260929 10:28:39.936295 28517 raft_consensus.cc:2454] T 0c1bd32ddcff42e28f83d1685a4a6e0d P a4679824ad5f48f4a46afdefd2c96a94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a5bf278a026465bad9f17660385bff0 in term 1.
I20260929 10:28:39.936312 28728 leader_election.cc:302] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 3a5bf278a026465bad9f17660385bff0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a5bf278a026465bad9f17660385bff0, 93dc1ff4066c4d21bd37e6241101305c; no voters:
I20260929 10:28:39.936481 28879 raft_consensus.cc:2781] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 3a5bf278a026465bad9f17660385bff0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:39.936702 28879 raft_consensus.cc:686] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 3a5bf278a026465bad9f17660385bff0 [term 1 LEADER]: Becoming Leader. State: Replica: 3a5bf278a026465bad9f17660385bff0, State: Running, Role: LEADER
I20260929 10:28:39.936978 28879 consensus_queue.cc:237] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 3a5bf278a026465bad9f17660385bff0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:39.939567 28359 catalog_manager.cc:5495] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 3a5bf278a026465bad9f17660385bff0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a5bf278a026465bad9f17660385bff0 (127.20.92.3). New cstate: current_term: 1 leader_uuid: "3a5bf278a026465bad9f17660385bff0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } health_report { overall_health: HEALTHY } } }
Sep 29 10:28:39 dist-test-slave-9cj5 krb5kdc[28315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677718, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.62@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM
I20260929 10:28:40.000233 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:39.987650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42018 (local address 127.20.92.62:39129)
0929 10:28:39.987887 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:39.987892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:39.988069 (+ 177us) server_negotiation.cc:407] Connection header received
0929 10:28:39.988156 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:39.988158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:39.988200 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:39.988270 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:39.988925 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.989548 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.990506 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.990756 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.990858 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:39.997124 (+ 6266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:39.997128 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:39.997130 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:39.997152 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:39.998512 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:39.999016 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:39.999018 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:39.999019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:39.999088 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:39.999350 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:39.999352 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:39.999353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:39.999524 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:39.999598 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:39.999960 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 10:28:40.000096 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":158}
I20260929 10:28:40.000269 28908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:39.987553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:39129 (local address 127.0.0.1:42018)
0929 10:28:39.987956 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:39.987970 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:39.988071 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:39.988311 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:39.988313 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:39.988687 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:39.988782 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:39.988788 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.989676 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.989679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:39.990330 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:39.990339 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:39.990899 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:39.990902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:39.990969 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:39.995476 (+ 4507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:39.995490 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:39.996946 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:39.998625 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:39.998633 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:39.998687 (+ 54us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:39.998921 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:39.999178 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:39.999180 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:39.999182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:39.999259 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:39.999642 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:39.999648 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:39.999801 (+ 153us) client_negotiation.cc:769] Sending connection context
0929 10:28:39.999910 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 10:28:40.000038 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":226,"threads_started":1}
I20260929 10:28:40.004676 28359 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:42018:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 10:28:40.005503 28359 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:28:40.010625 28497 tablet_service.cc:1467] Processing CreateTablet for tablet f79054a161cf4a11a125788fb64ff35f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=99e7e62f8fb34b2cb742c19678355f92]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 10:28:40.010730 28637 tablet_service.cc:1467] Processing CreateTablet for tablet f79054a161cf4a11a125788fb64ff35f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=99e7e62f8fb34b2cb742c19678355f92]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 10:28:40.011066 28497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f79054a161cf4a11a125788fb64ff35f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:40.010942 28777 tablet_service.cc:1467] Processing CreateTablet for tablet f79054a161cf4a11a125788fb64ff35f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=99e7e62f8fb34b2cb742c19678355f92]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 10:28:40.011106 28637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f79054a161cf4a11a125788fb64ff35f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:40.011271 28777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f79054a161cf4a11a125788fb64ff35f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:40.013628 28871 tablet_bootstrap.cc:492] T f79054a161cf4a11a125788fb64ff35f P 93dc1ff4066c4d21bd37e6241101305c: Bootstrap starting.
I20260929 10:28:40.013901 28872 tablet_bootstrap.cc:492] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94: Bootstrap starting.
I20260929 10:28:40.013943 28873 tablet_bootstrap.cc:492] T f79054a161cf4a11a125788fb64ff35f P 3a5bf278a026465bad9f17660385bff0: Bootstrap starting.
I20260929 10:28:40.014510 28871 tablet_bootstrap.cc:654] T f79054a161cf4a11a125788fb64ff35f P 93dc1ff4066c4d21bd37e6241101305c: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:40.014730 28873 tablet_bootstrap.cc:654] T f79054a161cf4a11a125788fb64ff35f P 3a5bf278a026465bad9f17660385bff0: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:40.014730 28872 tablet_bootstrap.cc:654] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:40.015918 28872 tablet_bootstrap.cc:492] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94: No bootstrap required, opened a new log
I20260929 10:28:40.015918 28873 tablet_bootstrap.cc:492] T f79054a161cf4a11a125788fb64ff35f P 3a5bf278a026465bad9f17660385bff0: No bootstrap required, opened a new log
I20260929 10:28:40.016005 28872 ts_tablet_manager.cc:1397] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:28:40.016011 28873 ts_tablet_manager.cc:1397] T f79054a161cf4a11a125788fb64ff35f P 3a5bf278a026465bad9f17660385bff0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:28:40.016428 28873 raft_consensus.cc:348] T f79054a161cf4a11a125788fb64ff35f P 3a5bf278a026465bad9f17660385bff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:40.016428 28872 raft_consensus.cc:348] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:40.016549 28873 raft_consensus.cc:374] T f79054a161cf4a11a125788fb64ff35f P 3a5bf278a026465bad9f17660385bff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:40.016549 28872 raft_consensus.cc:374] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:40.016585 28873 raft_consensus.cc:729] T f79054a161cf4a11a125788fb64ff35f P 3a5bf278a026465bad9f17660385bff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5bf278a026465bad9f17660385bff0, State: Initialized, Role: FOLLOWER
I20260929 10:28:40.016585 28872 raft_consensus.cc:729] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4679824ad5f48f4a46afdefd2c96a94, State: Initialized, Role: FOLLOWER
I20260929 10:28:40.016717 28872 consensus_queue.cc:260] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:40.016717 28873 consensus_queue.cc:260] T f79054a161cf4a11a125788fb64ff35f P 3a5bf278a026465bad9f17660385bff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:40.016981 28873 ts_tablet_manager.cc:1428] T f79054a161cf4a11a125788fb64ff35f P 3a5bf278a026465bad9f17660385bff0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:28:40.016981 28872 ts_tablet_manager.cc:1428] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 10:28:40.017184 28878 tablet_replica.cc:439] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } } }
I20260929 10:28:40.017323 28878 tablet_replica.cc:442] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:28:40.017495 28879 tablet_replica.cc:439] T f79054a161cf4a11a125788fb64ff35f P 3a5bf278a026465bad9f17660385bff0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } } }
I20260929 10:28:40.017625 28879 tablet_replica.cc:442] T f79054a161cf4a11a125788fb64ff35f P 3a5bf278a026465bad9f17660385bff0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:28:40.018385 28871 tablet_bootstrap.cc:492] T f79054a161cf4a11a125788fb64ff35f P 93dc1ff4066c4d21bd37e6241101305c: No bootstrap required, opened a new log
I20260929 10:28:40.018507 28871 ts_tablet_manager.cc:1397] T f79054a161cf4a11a125788fb64ff35f P 93dc1ff4066c4d21bd37e6241101305c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260929 10:28:40.018965 28871 raft_consensus.cc:348] T f79054a161cf4a11a125788fb64ff35f P 93dc1ff4066c4d21bd37e6241101305c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:40.019079 28871 raft_consensus.cc:374] T f79054a161cf4a11a125788fb64ff35f P 93dc1ff4066c4d21bd37e6241101305c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:40.019117 28871 raft_consensus.cc:729] T f79054a161cf4a11a125788fb64ff35f P 93dc1ff4066c4d21bd37e6241101305c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93dc1ff4066c4d21bd37e6241101305c, State: Initialized, Role: FOLLOWER
I20260929 10:28:40.019259 28871 consensus_queue.cc:260] T f79054a161cf4a11a125788fb64ff35f P 93dc1ff4066c4d21bd37e6241101305c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:40.019554 28871 ts_tablet_manager.cc:1428] T f79054a161cf4a11a125788fb64ff35f P 93dc1ff4066c4d21bd37e6241101305c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:28:40.019814 28877 tablet_replica.cc:439] T f79054a161cf4a11a125788fb64ff35f P 93dc1ff4066c4d21bd37e6241101305c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } } }
I20260929 10:28:40.019948 28877 tablet_replica.cc:442] T f79054a161cf4a11a125788fb64ff35f P 93dc1ff4066c4d21bd37e6241101305c: This TxnStatusTablet replica's current role is: FOLLOWER
W20260929 10:28:40.061439 28564 tablet.cc:2369] T 0c1bd32ddcff42e28f83d1685a4a6e0d P a4679824ad5f48f4a46afdefd2c96a94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:28:40.103555 28878 raft_consensus.cc:482] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:28:40.103718 28878 raft_consensus.cc:504] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:40.104173 28878 leader_election.cc:288] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93dc1ff4066c4d21bd37e6241101305c (127.20.92.2:44115), 3a5bf278a026465bad9f17660385bff0 (127.20.92.3:38451)
I20260929 10:28:40.104743 28796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f79054a161cf4a11a125788fb64ff35f" candidate_uuid: "a4679824ad5f48f4a46afdefd2c96a94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5bf278a026465bad9f17660385bff0" is_pre_election: true
I20260929 10:28:40.104708 28656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f79054a161cf4a11a125788fb64ff35f" candidate_uuid: "a4679824ad5f48f4a46afdefd2c96a94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93dc1ff4066c4d21bd37e6241101305c" is_pre_election: true
I20260929 10:28:40.104897 28796 raft_consensus.cc:2454] T f79054a161cf4a11a125788fb64ff35f P 3a5bf278a026465bad9f17660385bff0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4679824ad5f48f4a46afdefd2c96a94 in term 0.
I20260929 10:28:40.104897 28656 raft_consensus.cc:2454] T f79054a161cf4a11a125788fb64ff35f P 93dc1ff4066c4d21bd37e6241101305c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4679824ad5f48f4a46afdefd2c96a94 in term 0.
I20260929 10:28:40.105290 28448 leader_election.cc:302] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93dc1ff4066c4d21bd37e6241101305c, a4679824ad5f48f4a46afdefd2c96a94; no voters:
I20260929 10:28:40.105569 28878 raft_consensus.cc:2781] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:28:40.105636 28878 raft_consensus.cc:482] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:28:40.105679 28878 raft_consensus.cc:3037] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:40.106738 28878 raft_consensus.cc:504] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:40.107131 28878 leader_election.cc:288] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94 [CANDIDATE]: Term 1 election: Requested vote from peers 93dc1ff4066c4d21bd37e6241101305c (127.20.92.2:44115), 3a5bf278a026465bad9f17660385bff0 (127.20.92.3:38451)
I20260929 10:28:40.107453 28656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f79054a161cf4a11a125788fb64ff35f" candidate_uuid: "a4679824ad5f48f4a46afdefd2c96a94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93dc1ff4066c4d21bd37e6241101305c"
I20260929 10:28:40.107560 28656 raft_consensus.cc:3037] T f79054a161cf4a11a125788fb64ff35f P 93dc1ff4066c4d21bd37e6241101305c [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:40.107586 28796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f79054a161cf4a11a125788fb64ff35f" candidate_uuid: "a4679824ad5f48f4a46afdefd2c96a94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5bf278a026465bad9f17660385bff0"
I20260929 10:28:40.107713 28796 raft_consensus.cc:3037] T f79054a161cf4a11a125788fb64ff35f P 3a5bf278a026465bad9f17660385bff0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:40.108513 28656 raft_consensus.cc:2454] T f79054a161cf4a11a125788fb64ff35f P 93dc1ff4066c4d21bd37e6241101305c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4679824ad5f48f4a46afdefd2c96a94 in term 1.
I20260929 10:28:40.108748 28796 raft_consensus.cc:2454] T f79054a161cf4a11a125788fb64ff35f P 3a5bf278a026465bad9f17660385bff0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4679824ad5f48f4a46afdefd2c96a94 in term 1.
I20260929 10:28:40.108853 28448 leader_election.cc:302] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93dc1ff4066c4d21bd37e6241101305c, a4679824ad5f48f4a46afdefd2c96a94; no voters:
I20260929 10:28:40.109016 28878 raft_consensus.cc:2781] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:40.109127 28878 raft_consensus.cc:686] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94 [term 1 LEADER]: Becoming Leader. State: Replica: a4679824ad5f48f4a46afdefd2c96a94, State: Running, Role: LEADER
I20260929 10:28:40.109265 28878 consensus_queue.cc:237] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:40.109959 28882 tablet_replica.cc:439] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94: TxnStatusTablet state changed. Reason: New leader a4679824ad5f48f4a46afdefd2c96a94. Latest consensus state: current_term: 1 leader_uuid: "a4679824ad5f48f4a46afdefd2c96a94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } } }
I20260929 10:28:40.110077 28882 tablet_replica.cc:442] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94: This TxnStatusTablet replica's current role is: LEADER
I20260929 10:28:40.110357 28913 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 10:28:40.110522 28913 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 10:28:40.111075 28360 catalog_manager.cc:5495] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4679824ad5f48f4a46afdefd2c96a94 (127.20.92.1). New cstate: current_term: 1 leader_uuid: "a4679824ad5f48f4a46afdefd2c96a94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } health_report { overall_health: UNKNOWN } } }
I20260929 10:28:40.127038 28908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:40.122888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:46741 (local address 127.0.0.1:40458)
0929 10:28:40.123035 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:40.123052 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:40.123153 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:40.123475 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:40.123478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:40.123487 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:28:40.123582 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:40.123586 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.124664 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.124670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:40.125696 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:40.125706 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.126356 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.126359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:40.126449 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:40.126495 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:28:40.126780 (+ 285us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:28:40.126791 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 10:28:40.126842 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 10:28:40.126922 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 10:28:40.127050 28866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:40.123018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40458 (local address 127.20.92.1:46741)
0929 10:28:40.123129 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:40.123134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:40.123148 (+ 14us) server_negotiation.cc:407] Connection header received
0929 10:28:40.123275 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:40.123278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:40.123343 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:40.123426 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:28:40.123690 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.124475 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.125886 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.126241 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.126354 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:40.126613 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:28:40.126698 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:28:40.126770 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:40.126852 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 10:28:40.126910 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 10:28:40.129729 28656 raft_consensus.cc:1261] T f79054a161cf4a11a125788fb64ff35f P 93dc1ff4066c4d21bd37e6241101305c [term 1 FOLLOWER]: Refusing update from remote peer a4679824ad5f48f4a46afdefd2c96a94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:40.129724 28796 raft_consensus.cc:1261] T f79054a161cf4a11a125788fb64ff35f P 3a5bf278a026465bad9f17660385bff0 [term 1 FOLLOWER]: Refusing update from remote peer a4679824ad5f48f4a46afdefd2c96a94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:40.130204 28882 consensus_queue.cc:1035] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:28:40.130395 28878 consensus_queue.cc:1035] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:28:40.131724 28877 tablet_replica.cc:439] T f79054a161cf4a11a125788fb64ff35f P 93dc1ff4066c4d21bd37e6241101305c: TxnStatusTablet state changed. Reason: New leader a4679824ad5f48f4a46afdefd2c96a94. Latest consensus state: current_term: 1 leader_uuid: "a4679824ad5f48f4a46afdefd2c96a94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } } }
I20260929 10:28:40.131837 28877 tablet_replica.cc:442] T f79054a161cf4a11a125788fb64ff35f P 93dc1ff4066c4d21bd37e6241101305c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:28:40.132098 28879 tablet_replica.cc:439] T f79054a161cf4a11a125788fb64ff35f P 3a5bf278a026465bad9f17660385bff0: TxnStatusTablet state changed. Reason: New leader a4679824ad5f48f4a46afdefd2c96a94. Latest consensus state: current_term: 1 leader_uuid: "a4679824ad5f48f4a46afdefd2c96a94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } } }
I20260929 10:28:40.132207 28879 tablet_replica.cc:442] T f79054a161cf4a11a125788fb64ff35f P 3a5bf278a026465bad9f17660385bff0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:28:40.134222 28878 tablet_replica.cc:439] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a4679824ad5f48f4a46afdefd2c96a94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } } }
I20260929 10:28:40.134338 28878 tablet_replica.cc:442] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94: This TxnStatusTablet replica's current role is: LEADER
I20260929 10:28:40.135078 28877 tablet_replica.cc:439] T f79054a161cf4a11a125788fb64ff35f P 93dc1ff4066c4d21bd37e6241101305c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a4679824ad5f48f4a46afdefd2c96a94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } } }
I20260929 10:28:40.135186 28877 tablet_replica.cc:442] T f79054a161cf4a11a125788fb64ff35f P 93dc1ff4066c4d21bd37e6241101305c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:28:40.135792 28918 tablet_replica.cc:439] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a4679824ad5f48f4a46afdefd2c96a94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } } }
I20260929 10:28:40.135905 28918 tablet_replica.cc:442] T f79054a161cf4a11a125788fb64ff35f P a4679824ad5f48f4a46afdefd2c96a94: This TxnStatusTablet replica's current role is: LEADER
I20260929 10:28:40.137156 28879 tablet_replica.cc:439] T f79054a161cf4a11a125788fb64ff35f P 3a5bf278a026465bad9f17660385bff0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a4679824ad5f48f4a46afdefd2c96a94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } } }
I20260929 10:28:40.137272 28879 tablet_replica.cc:442] T f79054a161cf4a11a125788fb64ff35f P 3a5bf278a026465bad9f17660385bff0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:28:40.145489 28870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:40.141070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44954 (local address 127.20.92.3:38451)
0929 10:28:40.141210 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:40.141213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:40.141396 (+ 183us) server_negotiation.cc:407] Connection header received
0929 10:28:40.141490 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:40.141493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:40.141555 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:40.141676 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:28:40.142052 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.142655 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.143796 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.144066 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.144157 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:40.144473 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:28:40.145030 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:28:40.145161 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:40.145236 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 10:28:40.145289 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 10:28:40.158502 28866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:40.153372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:36723 (local address 127.20.92.1:46741)
0929 10:28:40.153511 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:40.153514 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:40.153648 (+ 134us) server_negotiation.cc:407] Connection header received
0929 10:28:40.153828 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:40.153831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:40.153922 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:40.154000 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:28:40.154283 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.154855 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.156658 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.156972 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.157251 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:40.157678 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:28:40.157942 (+ 264us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:28:40.158007 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:40.158315 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 10:28:40.158382 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 10:28:40.158546 28921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:40.153128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:46741 (local address 127.20.92.3:36723)
0929 10:28:40.153556 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:40.153572 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:40.153697 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:40.154033 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:40.154036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:40.154044 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:28:40.154174 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:40.154180 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.154966 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.154970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:40.156485 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:40.156498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.157275 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.157279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:40.157395 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:40.157461 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:28:40.158061 (+ 600us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:28:40.158074 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 10:28:40.158265 (+ 191us) client_negotiation.cc:240] Negotiation successful
0929 10:28:40.158363 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":108,"threads_started":1}
I20260929 10:28:40.166309 28656 raft_consensus.cc:1261] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 93dc1ff4066c4d21bd37e6241101305c [term 1 FOLLOWER]: Refusing update from remote peer 3a5bf278a026465bad9f17660385bff0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:40.166648 28517 raft_consensus.cc:1261] T 0c1bd32ddcff42e28f83d1685a4a6e0d P a4679824ad5f48f4a46afdefd2c96a94 [term 1 FOLLOWER]: Refusing update from remote peer 3a5bf278a026465bad9f17660385bff0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:40.166836 28879 consensus_queue.cc:1035] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 3a5bf278a026465bad9f17660385bff0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:28:40.167263 28879 consensus_queue.cc:1035] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 3a5bf278a026465bad9f17660385bff0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:28:40.194417 28870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:40.189543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:51891 (local address 127.20.92.3:38451)
0929 10:28:40.189669 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:40.189674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:40.189685 (+ 11us) server_negotiation.cc:407] Connection header received
0929 10:28:40.189889 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:40.189892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:40.189955 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:40.190009 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:28:40.190464 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.190975 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.192438 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.192830 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.192960 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:40.193678 (+ 718us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:28:40.193809 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:28:40.193889 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:40.194031 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 10:28:40.194089 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 10:28:40.194511 28927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:40.189254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:38451 (local address 127.20.92.1:51891)
0929 10:28:40.189589 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:40.189607 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:40.189716 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:40.190228 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:40.190235 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:40.190245 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:28:40.190360 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:40.190368 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.191100 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.191106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:40.192184 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:40.192213 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.192976 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.192980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:40.193251 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:40.193349 (+ 98us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:28:40.193913 (+ 564us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:28:40.193922 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 10:28:40.194000 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 10:28:40.194345 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":110,"threads_started":1}
I20260929 10:28:40.224313 28358 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:28:40.224494 28358 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:28:40.226666 28358 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4bcb22f0ca84d1e92a09724ab98d198: Sending DeleteTablet for 3 replicas of tablet 0c1bd32ddcff42e28f83d1685a4a6e0d
I20260929 10:28:40.227299 28497 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c1bd32ddcff42e28f83d1685a4a6e0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:40 UTC) from {username='oryx', principal='oryx/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:40434
I20260929 10:28:40.227532 28894 tablet_replica.cc:330] T 0c1bd32ddcff42e28f83d1685a4a6e0d P a4679824ad5f48f4a46afdefd2c96a94: stopping tablet replica
I20260929 10:28:40.227548 28637 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c1bd32ddcff42e28f83d1685a4a6e0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:40 UTC) from {username='oryx', principal='oryx/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:51392
I20260929 10:28:40.227663 28894 raft_consensus.cc:2229] T 0c1bd32ddcff42e28f83d1685a4a6e0d P a4679824ad5f48f4a46afdefd2c96a94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:28:40.227715 28895 tablet_replica.cc:330] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 93dc1ff4066c4d21bd37e6241101305c: stopping tablet replica
I20260929 10:28:40.227797 28894 raft_consensus.cc:2258] T 0c1bd32ddcff42e28f83d1685a4a6e0d P a4679824ad5f48f4a46afdefd2c96a94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:40.227808 28895 raft_consensus.cc:2229] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 93dc1ff4066c4d21bd37e6241101305c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:28:40.227921 28895 raft_consensus.cc:2258] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 93dc1ff4066c4d21bd37e6241101305c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:40.228662 28777 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c1bd32ddcff42e28f83d1685a4a6e0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:40 UTC) from {username='oryx', principal='oryx/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:44946
I20260929 10:28:40.229075 28895 ts_tablet_manager.cc:1905] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 93dc1ff4066c4d21bd37e6241101305c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:40.229498 28896 tablet_replica.cc:330] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 3a5bf278a026465bad9f17660385bff0: stopping tablet replica
I20260929 10:28:40.229614 28896 raft_consensus.cc:2229] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 3a5bf278a026465bad9f17660385bff0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:28:40.229971 28896 raft_consensus.cc:2258] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 3a5bf278a026465bad9f17660385bff0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:40.232354 28895 ts_tablet_manager.cc:1918] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 93dc1ff4066c4d21bd37e6241101305c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:28:40.232456 28895 log.cc:1196] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 93dc1ff4066c4d21bd37e6241101305c: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-1/wal/wals/0c1bd32ddcff42e28f83d1685a4a6e0d
I20260929 10:28:40.232698 28895 ts_tablet_manager.cc:1939] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 93dc1ff4066c4d21bd37e6241101305c: Deleting consensus metadata
I20260929 10:28:40.233464 28344 catalog_manager.cc:4841] TS 93dc1ff4066c4d21bd37e6241101305c (127.20.92.2:44115): tablet 0c1bd32ddcff42e28f83d1685a4a6e0d (table test-table [id=634d3ff3146642b8a4e095b1d7d29b19]) successfully deleted
I20260929 10:28:40.233811 28896 ts_tablet_manager.cc:1905] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 3a5bf278a026465bad9f17660385bff0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:40.236177 28896 ts_tablet_manager.cc:1918] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 3a5bf278a026465bad9f17660385bff0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:28:40.236282 28896 log.cc:1196] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 3a5bf278a026465bad9f17660385bff0: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-2/wal/wals/0c1bd32ddcff42e28f83d1685a4a6e0d
I20260929 10:28:40.236524 28896 ts_tablet_manager.cc:1939] T 0c1bd32ddcff42e28f83d1685a4a6e0d P 3a5bf278a026465bad9f17660385bff0: Deleting consensus metadata
I20260929 10:28:40.237563 28346 catalog_manager.cc:4841] TS 3a5bf278a026465bad9f17660385bff0 (127.20.92.3:38451): tablet 0c1bd32ddcff42e28f83d1685a4a6e0d (table test-table [id=634d3ff3146642b8a4e095b1d7d29b19]) successfully deleted
I20260929 10:28:40.241689 28894 ts_tablet_manager.cc:1905] T 0c1bd32ddcff42e28f83d1685a4a6e0d P a4679824ad5f48f4a46afdefd2c96a94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:40.245383 28894 ts_tablet_manager.cc:1918] T 0c1bd32ddcff42e28f83d1685a4a6e0d P a4679824ad5f48f4a46afdefd2c96a94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:28:40.245476 28894 log.cc:1196] T 0c1bd32ddcff42e28f83d1685a4a6e0d P a4679824ad5f48f4a46afdefd2c96a94: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-0/wal/wals/0c1bd32ddcff42e28f83d1685a4a6e0d
I20260929 10:28:40.245965 28894 ts_tablet_manager.cc:1939] T 0c1bd32ddcff42e28f83d1685a4a6e0d P a4679824ad5f48f4a46afdefd2c96a94: Deleting consensus metadata
I20260929 10:28:40.246935 28345 catalog_manager.cc:4841] TS a4679824ad5f48f4a46afdefd2c96a94 (127.20.92.1:46741): tablet 0c1bd32ddcff42e28f83d1685a4a6e0d (table test-table [id=634d3ff3146642b8a4e095b1d7d29b19]) successfully deleted
Sep 29 10:28:40 dist-test-slave-9cj5 krb5kdc[28315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 10:28:40 dist-test-slave-9cj5 krb5kdc[28315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:40 dist-test-slave-9cj5 krb5kdc[28315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 10:28:40.267717 28409 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:40.260124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42026 (local address 127.20.92.62:39129)
0929 10:28:40.260283 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:40.260289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:40.260572 (+ 283us) server_negotiation.cc:407] Connection header received
0929 10:28:40.260820 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:40.260824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:40.260879 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:40.260942 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:40.261958 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.262542 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.263534 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.263786 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.263913 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:40.267551 (+ 3638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42026: BlockingRecv error: recv got EOF from 127.0.0.1:42026 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 29 10:28:40 dist-test-slave-9cj5 krb5kdc[28315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM
I20260929 10:28:40.280189 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:40.269735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42028 (local address 127.20.92.62:39129)
0929 10:28:40.269904 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:40.269911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:40.270423 (+ 512us) server_negotiation.cc:407] Connection header received
0929 10:28:40.270588 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:40.270592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:40.270653 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:40.270748 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:40.271617 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.272219 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.273085 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.273339 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.273461 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:40.276149 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:40.276154 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:40.276157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:40.276189 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:40.278196 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:40.278743 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:40.278747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:40.278750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:40.278834 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:40.279127 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:40.279130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:40.279132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:40.279357 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:40.279465 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:40.279868 (+ 403us) server_negotiation.cc:299] Negotiation successful
0929 10:28:40.280003 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 10:28:40.283715 28358 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:28:40.284222 28358 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:28:40.289444 28497 tablet_service.cc:1467] Processing CreateTablet for tablet 6e1210de371746d1b38f831d523219f3 (DEFAULT_TABLE table=test-table [id=f5a5758e683a4b81a7bd75ddac938ba6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:40.290088 28497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e1210de371746d1b38f831d523219f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:40.290231 28637 tablet_service.cc:1467] Processing CreateTablet for tablet 6e1210de371746d1b38f831d523219f3 (DEFAULT_TABLE table=test-table [id=f5a5758e683a4b81a7bd75ddac938ba6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:40.290501 28637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e1210de371746d1b38f831d523219f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:40.292959 28871 tablet_bootstrap.cc:492] T 6e1210de371746d1b38f831d523219f3 P 93dc1ff4066c4d21bd37e6241101305c: Bootstrap starting.
I20260929 10:28:40.293885 28871 tablet_bootstrap.cc:654] T 6e1210de371746d1b38f831d523219f3 P 93dc1ff4066c4d21bd37e6241101305c: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:40.294837 28872 tablet_bootstrap.cc:492] T 6e1210de371746d1b38f831d523219f3 P a4679824ad5f48f4a46afdefd2c96a94: Bootstrap starting.
I20260929 10:28:40.295564 28872 tablet_bootstrap.cc:654] T 6e1210de371746d1b38f831d523219f3 P a4679824ad5f48f4a46afdefd2c96a94: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:40.296653 28777 tablet_service.cc:1467] Processing CreateTablet for tablet 6e1210de371746d1b38f831d523219f3 (DEFAULT_TABLE table=test-table [id=f5a5758e683a4b81a7bd75ddac938ba6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:40.296828 28872 tablet_bootstrap.cc:492] T 6e1210de371746d1b38f831d523219f3 P a4679824ad5f48f4a46afdefd2c96a94: No bootstrap required, opened a new log
I20260929 10:28:40.296912 28872 ts_tablet_manager.cc:1397] T 6e1210de371746d1b38f831d523219f3 P a4679824ad5f48f4a46afdefd2c96a94: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:28:40.296937 28777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e1210de371746d1b38f831d523219f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:40.297360 28872 raft_consensus.cc:348] T 6e1210de371746d1b38f831d523219f3 P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:40.297474 28872 raft_consensus.cc:374] T 6e1210de371746d1b38f831d523219f3 P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:40.297506 28872 raft_consensus.cc:729] T 6e1210de371746d1b38f831d523219f3 P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4679824ad5f48f4a46afdefd2c96a94, State: Initialized, Role: FOLLOWER
I20260929 10:28:40.297629 28872 consensus_queue.cc:260] T 6e1210de371746d1b38f831d523219f3 P a4679824ad5f48f4a46afdefd2c96a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:40.297902 28872 ts_tablet_manager.cc:1428] T 6e1210de371746d1b38f831d523219f3 P a4679824ad5f48f4a46afdefd2c96a94: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:28:40.299322 28871 tablet_bootstrap.cc:492] T 6e1210de371746d1b38f831d523219f3 P 93dc1ff4066c4d21bd37e6241101305c: No bootstrap required, opened a new log
I20260929 10:28:40.299412 28871 ts_tablet_manager.cc:1397] T 6e1210de371746d1b38f831d523219f3 P 93dc1ff4066c4d21bd37e6241101305c: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20260929 10:28:40.299935 28873 tablet_bootstrap.cc:492] T 6e1210de371746d1b38f831d523219f3 P 3a5bf278a026465bad9f17660385bff0: Bootstrap starting.
I20260929 10:28:40.300318 28871 raft_consensus.cc:348] T 6e1210de371746d1b38f831d523219f3 P 93dc1ff4066c4d21bd37e6241101305c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:40.300469 28871 raft_consensus.cc:374] T 6e1210de371746d1b38f831d523219f3 P 93dc1ff4066c4d21bd37e6241101305c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:40.300503 28871 raft_consensus.cc:729] T 6e1210de371746d1b38f831d523219f3 P 93dc1ff4066c4d21bd37e6241101305c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93dc1ff4066c4d21bd37e6241101305c, State: Initialized, Role: FOLLOWER
I20260929 10:28:40.301000 28873 tablet_bootstrap.cc:654] T 6e1210de371746d1b38f831d523219f3 P 3a5bf278a026465bad9f17660385bff0: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:40.302024 28871 consensus_queue.cc:260] T 6e1210de371746d1b38f831d523219f3 P 93dc1ff4066c4d21bd37e6241101305c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:40.302978 28871 ts_tablet_manager.cc:1428] T 6e1210de371746d1b38f831d523219f3 P 93dc1ff4066c4d21bd37e6241101305c: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260929 10:28:40.311910 28873 tablet_bootstrap.cc:492] T 6e1210de371746d1b38f831d523219f3 P 3a5bf278a026465bad9f17660385bff0: No bootstrap required, opened a new log
I20260929 10:28:40.312024 28873 ts_tablet_manager.cc:1397] T 6e1210de371746d1b38f831d523219f3 P 3a5bf278a026465bad9f17660385bff0: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260929 10:28:40.312485 28873 raft_consensus.cc:348] T 6e1210de371746d1b38f831d523219f3 P 3a5bf278a026465bad9f17660385bff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:40.312613 28873 raft_consensus.cc:374] T 6e1210de371746d1b38f831d523219f3 P 3a5bf278a026465bad9f17660385bff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:40.312644 28873 raft_consensus.cc:729] T 6e1210de371746d1b38f831d523219f3 P 3a5bf278a026465bad9f17660385bff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5bf278a026465bad9f17660385bff0, State: Initialized, Role: FOLLOWER
I20260929 10:28:40.312783 28873 consensus_queue.cc:260] T 6e1210de371746d1b38f831d523219f3 P 3a5bf278a026465bad9f17660385bff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:40.313056 28873 ts_tablet_manager.cc:1428] T 6e1210de371746d1b38f831d523219f3 P 3a5bf278a026465bad9f17660385bff0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 10:28:40.430267 28877 raft_consensus.cc:482] T 6e1210de371746d1b38f831d523219f3 P 93dc1ff4066c4d21bd37e6241101305c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:28:40.430414 28877 raft_consensus.cc:504] T 6e1210de371746d1b38f831d523219f3 P 93dc1ff4066c4d21bd37e6241101305c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:40.431628 28877 leader_election.cc:288] T 6e1210de371746d1b38f831d523219f3 P 93dc1ff4066c4d21bd37e6241101305c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4679824ad5f48f4a46afdefd2c96a94 (127.20.92.1:46741), 3a5bf278a026465bad9f17660385bff0 (127.20.92.3:38451)
I20260929 10:28:40.435629 28870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:40.432230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:36253 (local address 127.20.92.3:38451)
0929 10:28:40.432351 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:40.432354 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:40.432365 (+ 11us) server_negotiation.cc:407] Connection header received
0929 10:28:40.432429 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:40.432433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:40.432479 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:40.432554 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:40.432814 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.433348 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.434642 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.435040 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.435151 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:40.435227 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:40.435407 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 10:28:40.435486 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 10:28:40.435652 28866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:40.431930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:47923 (local address 127.20.92.1:46741)
0929 10:28:40.432061 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:40.432066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:40.432162 (+ 96us) server_negotiation.cc:407] Connection header received
0929 10:28:40.432307 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:40.432313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:40.432369 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:40.432427 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:28:40.432902 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.433387 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.434674 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.435057 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.435182 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:40.435244 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:40.435465 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 10:28:40.435546 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 10:28:40.435667 28946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:40.431888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:38451 (local address 127.20.92.2:36253)
0929 10:28:40.432115 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:40.432131 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:40.432224 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:40.432590 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:40.432592 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:40.432607 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:40.432677 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:40.432681 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.433503 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.433507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:40.434515 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:40.434525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.435172 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.435176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:40.435266 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:40.435296 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 10:28:40.435363 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 10:28:40.435456 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":39,"thread_start_us":81,"threads_started":1}
I20260929 10:28:40.435665 28945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:40.431808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:46741 (local address 127.20.92.2:47923)
0929 10:28:40.432061 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:40.432077 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:40.432194 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:40.432461 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:40.432464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:40.432481 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:28:40.432615 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:40.432623 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.433534 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.433537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:40.434519 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:40.434528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.435226 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.435230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:40.435317 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:40.435334 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 10:28:40.435436 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 10:28:40.435500 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":67,"thread_start_us":93,"threads_started":1}
I20260929 10:28:40.436249 28796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e1210de371746d1b38f831d523219f3" candidate_uuid: "93dc1ff4066c4d21bd37e6241101305c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5bf278a026465bad9f17660385bff0" is_pre_election: true
I20260929 10:28:40.436347 28517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e1210de371746d1b38f831d523219f3" candidate_uuid: "93dc1ff4066c4d21bd37e6241101305c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4679824ad5f48f4a46afdefd2c96a94" is_pre_election: true
I20260929 10:28:40.436419 28796 raft_consensus.cc:2454] T 6e1210de371746d1b38f831d523219f3 P 3a5bf278a026465bad9f17660385bff0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93dc1ff4066c4d21bd37e6241101305c in term 0.
I20260929 10:28:40.436511 28517 raft_consensus.cc:2454] T 6e1210de371746d1b38f831d523219f3 P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93dc1ff4066c4d21bd37e6241101305c in term 0.
I20260929 10:28:40.436800 28590 leader_election.cc:302] T 6e1210de371746d1b38f831d523219f3 P 93dc1ff4066c4d21bd37e6241101305c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a5bf278a026465bad9f17660385bff0, 93dc1ff4066c4d21bd37e6241101305c; no voters:
I20260929 10:28:40.437063 28877 raft_consensus.cc:2781] T 6e1210de371746d1b38f831d523219f3 P 93dc1ff4066c4d21bd37e6241101305c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:28:40.437143 28877 raft_consensus.cc:482] T 6e1210de371746d1b38f831d523219f3 P 93dc1ff4066c4d21bd37e6241101305c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:28:40.437206 28877 raft_consensus.cc:3037] T 6e1210de371746d1b38f831d523219f3 P 93dc1ff4066c4d21bd37e6241101305c [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:40.438279 28877 raft_consensus.cc:504] T 6e1210de371746d1b38f831d523219f3 P 93dc1ff4066c4d21bd37e6241101305c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:40.438670 28877 leader_election.cc:288] T 6e1210de371746d1b38f831d523219f3 P 93dc1ff4066c4d21bd37e6241101305c [CANDIDATE]: Term 1 election: Requested vote from peers a4679824ad5f48f4a46afdefd2c96a94 (127.20.92.1:46741), 3a5bf278a026465bad9f17660385bff0 (127.20.92.3:38451)
I20260929 10:28:40.438997 28517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e1210de371746d1b38f831d523219f3" candidate_uuid: "93dc1ff4066c4d21bd37e6241101305c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4679824ad5f48f4a46afdefd2c96a94"
I20260929 10:28:40.439061 28796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e1210de371746d1b38f831d523219f3" candidate_uuid: "93dc1ff4066c4d21bd37e6241101305c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5bf278a026465bad9f17660385bff0"
I20260929 10:28:40.439143 28517 raft_consensus.cc:3037] T 6e1210de371746d1b38f831d523219f3 P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:40.439249 28796 raft_consensus.cc:3037] T 6e1210de371746d1b38f831d523219f3 P 3a5bf278a026465bad9f17660385bff0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:40.440138 28517 raft_consensus.cc:2454] T 6e1210de371746d1b38f831d523219f3 P a4679824ad5f48f4a46afdefd2c96a94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93dc1ff4066c4d21bd37e6241101305c in term 1.
I20260929 10:28:40.440301 28796 raft_consensus.cc:2454] T 6e1210de371746d1b38f831d523219f3 P 3a5bf278a026465bad9f17660385bff0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93dc1ff4066c4d21bd37e6241101305c in term 1.
I20260929 10:28:40.440472 28589 leader_election.cc:302] T 6e1210de371746d1b38f831d523219f3 P 93dc1ff4066c4d21bd37e6241101305c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93dc1ff4066c4d21bd37e6241101305c, a4679824ad5f48f4a46afdefd2c96a94; no voters:
I20260929 10:28:40.440637 28877 raft_consensus.cc:2781] T 6e1210de371746d1b38f831d523219f3 P 93dc1ff4066c4d21bd37e6241101305c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:40.440896 28877 raft_consensus.cc:686] T 6e1210de371746d1b38f831d523219f3 P 93dc1ff4066c4d21bd37e6241101305c [term 1 LEADER]: Becoming Leader. State: Replica: 93dc1ff4066c4d21bd37e6241101305c, State: Running, Role: LEADER
I20260929 10:28:40.441200 28877 consensus_queue.cc:237] T 6e1210de371746d1b38f831d523219f3 P 93dc1ff4066c4d21bd37e6241101305c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:40.443620 28358 catalog_manager.cc:5495] T 6e1210de371746d1b38f831d523219f3 P 93dc1ff4066c4d21bd37e6241101305c reported cstate change: term changed from 0 to 1, leader changed from <none> to 93dc1ff4066c4d21bd37e6241101305c (127.20.92.2). New cstate: current_term: 1 leader_uuid: "93dc1ff4066c4d21bd37e6241101305c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } health_report { overall_health: UNKNOWN } } }
I20260929 10:28:40.467854 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:40.463838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51404 (local address 127.20.92.2:44115)
0929 10:28:40.464146 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:40.464157 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:40.464172 (+ 15us) server_negotiation.cc:407] Connection header received
0929 10:28:40.464262 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:40.464264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:40.464321 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:40.464450 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:28:40.464700 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.465266 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.466191 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.466441 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.466549 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:40.466840 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:28:40.467289 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:28:40.467442 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:40.467550 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 10:28:40.467600 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":104,"threads_started":1}
I20260929 10:28:40.470613 28517 raft_consensus.cc:1261] T 6e1210de371746d1b38f831d523219f3 P a4679824ad5f48f4a46afdefd2c96a94 [term 1 FOLLOWER]: Refusing update from remote peer 93dc1ff4066c4d21bd37e6241101305c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:40.470613 28796 raft_consensus.cc:1261] T 6e1210de371746d1b38f831d523219f3 P 3a5bf278a026465bad9f17660385bff0 [term 1 FOLLOWER]: Refusing update from remote peer 93dc1ff4066c4d21bd37e6241101305c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:40.471132 28877 consensus_queue.cc:1035] T 6e1210de371746d1b38f831d523219f3 P 93dc1ff4066c4d21bd37e6241101305c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:28:40.471376 28947 consensus_queue.cc:1035] T 6e1210de371746d1b38f831d523219f3 P 93dc1ff4066c4d21bd37e6241101305c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:28:40.482678 28358 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:28:40.482863 28358 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:28:40.484905 28358 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4bcb22f0ca84d1e92a09724ab98d198: Sending DeleteTablet for 3 replicas of tablet 6e1210de371746d1b38f831d523219f3
I20260929 10:28:40.485442 28497 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e1210de371746d1b38f831d523219f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:40 UTC) from {username='oryx', principal='oryx/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:40434
I20260929 10:28:40.485575 28637 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e1210de371746d1b38f831d523219f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:40 UTC) from {username='oryx', principal='oryx/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:51392
I20260929 10:28:40.485656 28894 tablet_replica.cc:330] T 6e1210de371746d1b38f831d523219f3 P a4679824ad5f48f4a46afdefd2c96a94: stopping tablet replica
I20260929 10:28:40.485805 28894 raft_consensus.cc:2229] T 6e1210de371746d1b38f831d523219f3 P a4679824ad5f48f4a46afdefd2c96a94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:28:40.485857 28895 tablet_replica.cc:330] T 6e1210de371746d1b38f831d523219f3 P 93dc1ff4066c4d21bd37e6241101305c: stopping tablet replica
I20260929 10:28:40.485947 28894 raft_consensus.cc:2258] T 6e1210de371746d1b38f831d523219f3 P a4679824ad5f48f4a46afdefd2c96a94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:40.485988 28895 raft_consensus.cc:2229] T 6e1210de371746d1b38f831d523219f3 P 93dc1ff4066c4d21bd37e6241101305c [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:28:40.486277 28895 raft_consensus.cc:2258] T 6e1210de371746d1b38f831d523219f3 P 93dc1ff4066c4d21bd37e6241101305c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:40.486723 28894 ts_tablet_manager.cc:1905] T 6e1210de371746d1b38f831d523219f3 P a4679824ad5f48f4a46afdefd2c96a94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:40.486766 28777 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e1210de371746d1b38f831d523219f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:40 UTC) from {username='oryx', principal='oryx/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:44946
I20260929 10:28:40.486963 28895 ts_tablet_manager.cc:1905] T 6e1210de371746d1b38f831d523219f3 P 93dc1ff4066c4d21bd37e6241101305c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:40.489223 28894 ts_tablet_manager.cc:1918] T 6e1210de371746d1b38f831d523219f3 P a4679824ad5f48f4a46afdefd2c96a94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:28:40.489223 28895 ts_tablet_manager.cc:1918] T 6e1210de371746d1b38f831d523219f3 P 93dc1ff4066c4d21bd37e6241101305c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:28:40.489295 28895 log.cc:1196] T 6e1210de371746d1b38f831d523219f3 P 93dc1ff4066c4d21bd37e6241101305c: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-1/wal/wals/6e1210de371746d1b38f831d523219f3
I20260929 10:28:40.489295 28894 log.cc:1196] T 6e1210de371746d1b38f831d523219f3 P a4679824ad5f48f4a46afdefd2c96a94: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-0/wal/wals/6e1210de371746d1b38f831d523219f3
I20260929 10:28:40.489554 28895 ts_tablet_manager.cc:1939] T 6e1210de371746d1b38f831d523219f3 P 93dc1ff4066c4d21bd37e6241101305c: Deleting consensus metadata
I20260929 10:28:40.489554 28894 ts_tablet_manager.cc:1939] T 6e1210de371746d1b38f831d523219f3 P a4679824ad5f48f4a46afdefd2c96a94: Deleting consensus metadata
I20260929 10:28:40.489784 28896 tablet_replica.cc:330] T 6e1210de371746d1b38f831d523219f3 P 3a5bf278a026465bad9f17660385bff0: stopping tablet replica
I20260929 10:28:40.489979 28896 raft_consensus.cc:2229] T 6e1210de371746d1b38f831d523219f3 P 3a5bf278a026465bad9f17660385bff0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:28:40.490106 28896 raft_consensus.cc:2258] T 6e1210de371746d1b38f831d523219f3 P 3a5bf278a026465bad9f17660385bff0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:40.490218 28359 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 10:28:40.490401 28344 catalog_manager.cc:4841] TS 93dc1ff4066c4d21bd37e6241101305c (127.20.92.2:44115): tablet 6e1210de371746d1b38f831d523219f3 (table test-table [id=f5a5758e683a4b81a7bd75ddac938ba6]) successfully deleted
W20260929 10:28:40.490687 28359 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:28:40.491210 28896 ts_tablet_manager.cc:1905] T 6e1210de371746d1b38f831d523219f3 P 3a5bf278a026465bad9f17660385bff0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:40.491961 28345 catalog_manager.cc:4841] TS a4679824ad5f48f4a46afdefd2c96a94 (127.20.92.1:46741): tablet 6e1210de371746d1b38f831d523219f3 (table test-table [id=f5a5758e683a4b81a7bd75ddac938ba6]) successfully deleted
I20260929 10:28:40.493319 28896 ts_tablet_manager.cc:1918] T 6e1210de371746d1b38f831d523219f3 P 3a5bf278a026465bad9f17660385bff0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:28:40.493414 28896 log.cc:1196] T 6e1210de371746d1b38f831d523219f3 P 3a5bf278a026465bad9f17660385bff0: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-2/wal/wals/6e1210de371746d1b38f831d523219f3
I20260929 10:28:40.493641 28896 ts_tablet_manager.cc:1939] T 6e1210de371746d1b38f831d523219f3 P 3a5bf278a026465bad9f17660385bff0: Deleting consensus metadata
I20260929 10:28:40.494771 28346 catalog_manager.cc:4841] TS 3a5bf278a026465bad9f17660385bff0 (127.20.92.3:38451): tablet 6e1210de371746d1b38f831d523219f3 (table test-table [id=f5a5758e683a4b81a7bd75ddac938ba6]) successfully deleted
I20260929 10:28:40.498021 28497 tablet_service.cc:1467] Processing CreateTablet for tablet c45ab7e9a9bc4f9dbed21bc79dd3832c (DEFAULT_TABLE table=test-table [id=bbccc1e669a64eed89eaa66d2123f5a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:40.498162 28777 tablet_service.cc:1467] Processing CreateTablet for tablet c45ab7e9a9bc4f9dbed21bc79dd3832c (DEFAULT_TABLE table=test-table [id=bbccc1e669a64eed89eaa66d2123f5a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:40.498293 28497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c45ab7e9a9bc4f9dbed21bc79dd3832c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:40.498409 28777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c45ab7e9a9bc4f9dbed21bc79dd3832c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:40.500566 28873 tablet_bootstrap.cc:492] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 3a5bf278a026465bad9f17660385bff0: Bootstrap starting.
I20260929 10:28:40.501331 28873 tablet_bootstrap.cc:654] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 3a5bf278a026465bad9f17660385bff0: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:40.501894 28872 tablet_bootstrap.cc:492] T c45ab7e9a9bc4f9dbed21bc79dd3832c P a4679824ad5f48f4a46afdefd2c96a94: Bootstrap starting.
I20260929 10:28:40.501981 28637 tablet_service.cc:1467] Processing CreateTablet for tablet c45ab7e9a9bc4f9dbed21bc79dd3832c (DEFAULT_TABLE table=test-table [id=bbccc1e669a64eed89eaa66d2123f5a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:28:40.502241 28637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c45ab7e9a9bc4f9dbed21bc79dd3832c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:40.502666 28872 tablet_bootstrap.cc:654] T c45ab7e9a9bc4f9dbed21bc79dd3832c P a4679824ad5f48f4a46afdefd2c96a94: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:40.503921 28872 tablet_bootstrap.cc:492] T c45ab7e9a9bc4f9dbed21bc79dd3832c P a4679824ad5f48f4a46afdefd2c96a94: No bootstrap required, opened a new log
I20260929 10:28:40.503991 28872 ts_tablet_manager.cc:1397] T c45ab7e9a9bc4f9dbed21bc79dd3832c P a4679824ad5f48f4a46afdefd2c96a94: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:28:40.504410 28872 raft_consensus.cc:348] T c45ab7e9a9bc4f9dbed21bc79dd3832c P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:40.504521 28872 raft_consensus.cc:374] T c45ab7e9a9bc4f9dbed21bc79dd3832c P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:40.504554 28872 raft_consensus.cc:729] T c45ab7e9a9bc4f9dbed21bc79dd3832c P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4679824ad5f48f4a46afdefd2c96a94, State: Initialized, Role: FOLLOWER
I20260929 10:28:40.504642 28871 tablet_bootstrap.cc:492] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 93dc1ff4066c4d21bd37e6241101305c: Bootstrap starting.
I20260929 10:28:40.504678 28872 consensus_queue.cc:260] T c45ab7e9a9bc4f9dbed21bc79dd3832c P a4679824ad5f48f4a46afdefd2c96a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:40.504966 28872 ts_tablet_manager.cc:1428] T c45ab7e9a9bc4f9dbed21bc79dd3832c P a4679824ad5f48f4a46afdefd2c96a94: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:28:40.505566 28871 tablet_bootstrap.cc:654] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 93dc1ff4066c4d21bd37e6241101305c: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:40.506321 28873 tablet_bootstrap.cc:492] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 3a5bf278a026465bad9f17660385bff0: No bootstrap required, opened a new log
I20260929 10:28:40.506410 28873 ts_tablet_manager.cc:1397] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 3a5bf278a026465bad9f17660385bff0: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260929 10:28:40.506769 28873 raft_consensus.cc:348] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 3a5bf278a026465bad9f17660385bff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:40.506892 28873 raft_consensus.cc:374] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 3a5bf278a026465bad9f17660385bff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:40.506923 28873 raft_consensus.cc:729] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 3a5bf278a026465bad9f17660385bff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5bf278a026465bad9f17660385bff0, State: Initialized, Role: FOLLOWER
I20260929 10:28:40.507045 28873 consensus_queue.cc:260] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 3a5bf278a026465bad9f17660385bff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:40.507304 28873 ts_tablet_manager.cc:1428] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 3a5bf278a026465bad9f17660385bff0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:28:40.519057 28871 tablet_bootstrap.cc:492] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 93dc1ff4066c4d21bd37e6241101305c: No bootstrap required, opened a new log
I20260929 10:28:40.519176 28871 ts_tablet_manager.cc:1397] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 93dc1ff4066c4d21bd37e6241101305c: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.002s
I20260929 10:28:40.519639 28871 raft_consensus.cc:348] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 93dc1ff4066c4d21bd37e6241101305c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:40.519761 28871 raft_consensus.cc:374] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 93dc1ff4066c4d21bd37e6241101305c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:40.519791 28871 raft_consensus.cc:729] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 93dc1ff4066c4d21bd37e6241101305c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93dc1ff4066c4d21bd37e6241101305c, State: Initialized, Role: FOLLOWER
I20260929 10:28:40.519938 28871 consensus_queue.cc:260] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 93dc1ff4066c4d21bd37e6241101305c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:40.520213 28871 ts_tablet_manager.cc:1428] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 93dc1ff4066c4d21bd37e6241101305c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 10:28:40.562712 28882 raft_consensus.cc:482] T c45ab7e9a9bc4f9dbed21bc79dd3832c P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:28:40.562875 28882 raft_consensus.cc:504] T c45ab7e9a9bc4f9dbed21bc79dd3832c P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:40.563333 28882 leader_election.cc:288] T c45ab7e9a9bc4f9dbed21bc79dd3832c P a4679824ad5f48f4a46afdefd2c96a94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93dc1ff4066c4d21bd37e6241101305c (127.20.92.2:44115), 3a5bf278a026465bad9f17660385bff0 (127.20.92.3:38451)
I20260929 10:28:40.563841 28656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45ab7e9a9bc4f9dbed21bc79dd3832c" candidate_uuid: "a4679824ad5f48f4a46afdefd2c96a94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93dc1ff4066c4d21bd37e6241101305c" is_pre_election: true
I20260929 10:28:40.563867 28796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45ab7e9a9bc4f9dbed21bc79dd3832c" candidate_uuid: "a4679824ad5f48f4a46afdefd2c96a94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5bf278a026465bad9f17660385bff0" is_pre_election: true
I20260929 10:28:40.564002 28796 raft_consensus.cc:2454] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 3a5bf278a026465bad9f17660385bff0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4679824ad5f48f4a46afdefd2c96a94 in term 0.
I20260929 10:28:40.564002 28656 raft_consensus.cc:2454] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 93dc1ff4066c4d21bd37e6241101305c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4679824ad5f48f4a46afdefd2c96a94 in term 0.
I20260929 10:28:40.564304 28448 leader_election.cc:302] T c45ab7e9a9bc4f9dbed21bc79dd3832c P a4679824ad5f48f4a46afdefd2c96a94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93dc1ff4066c4d21bd37e6241101305c, a4679824ad5f48f4a46afdefd2c96a94; no voters:
I20260929 10:28:40.564530 28882 raft_consensus.cc:2781] T c45ab7e9a9bc4f9dbed21bc79dd3832c P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:28:40.564599 28882 raft_consensus.cc:482] T c45ab7e9a9bc4f9dbed21bc79dd3832c P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:28:40.564638 28882 raft_consensus.cc:3037] T c45ab7e9a9bc4f9dbed21bc79dd3832c P a4679824ad5f48f4a46afdefd2c96a94 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:40.565958 28882 raft_consensus.cc:504] T c45ab7e9a9bc4f9dbed21bc79dd3832c P a4679824ad5f48f4a46afdefd2c96a94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:40.566313 28882 leader_election.cc:288] T c45ab7e9a9bc4f9dbed21bc79dd3832c P a4679824ad5f48f4a46afdefd2c96a94 [CANDIDATE]: Term 1 election: Requested vote from peers 93dc1ff4066c4d21bd37e6241101305c (127.20.92.2:44115), 3a5bf278a026465bad9f17660385bff0 (127.20.92.3:38451)
I20260929 10:28:40.566666 28656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45ab7e9a9bc4f9dbed21bc79dd3832c" candidate_uuid: "a4679824ad5f48f4a46afdefd2c96a94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93dc1ff4066c4d21bd37e6241101305c"
I20260929 10:28:40.566694 28796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45ab7e9a9bc4f9dbed21bc79dd3832c" candidate_uuid: "a4679824ad5f48f4a46afdefd2c96a94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5bf278a026465bad9f17660385bff0"
I20260929 10:28:40.566810 28796 raft_consensus.cc:3037] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 3a5bf278a026465bad9f17660385bff0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:40.566839 28656 raft_consensus.cc:3037] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 93dc1ff4066c4d21bd37e6241101305c [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:40.567945 28796 raft_consensus.cc:2454] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 3a5bf278a026465bad9f17660385bff0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4679824ad5f48f4a46afdefd2c96a94 in term 1.
I20260929 10:28:40.568028 28656 raft_consensus.cc:2454] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 93dc1ff4066c4d21bd37e6241101305c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4679824ad5f48f4a46afdefd2c96a94 in term 1.
I20260929 10:28:40.568255 28450 leader_election.cc:302] T c45ab7e9a9bc4f9dbed21bc79dd3832c P a4679824ad5f48f4a46afdefd2c96a94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a5bf278a026465bad9f17660385bff0, a4679824ad5f48f4a46afdefd2c96a94; no voters:
I20260929 10:28:40.568531 28882 raft_consensus.cc:2781] T c45ab7e9a9bc4f9dbed21bc79dd3832c P a4679824ad5f48f4a46afdefd2c96a94 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:40.568655 28882 raft_consensus.cc:686] T c45ab7e9a9bc4f9dbed21bc79dd3832c P a4679824ad5f48f4a46afdefd2c96a94 [term 1 LEADER]: Becoming Leader. State: Replica: a4679824ad5f48f4a46afdefd2c96a94, State: Running, Role: LEADER
I20260929 10:28:40.568792 28882 consensus_queue.cc:237] T c45ab7e9a9bc4f9dbed21bc79dd3832c P a4679824ad5f48f4a46afdefd2c96a94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } }
I20260929 10:28:40.570554 28358 catalog_manager.cc:5495] T c45ab7e9a9bc4f9dbed21bc79dd3832c P a4679824ad5f48f4a46afdefd2c96a94 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4679824ad5f48f4a46afdefd2c96a94 (127.20.92.1). New cstate: current_term: 1 leader_uuid: "a4679824ad5f48f4a46afdefd2c96a94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4679824ad5f48f4a46afdefd2c96a94" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 } health_report { overall_health: UNKNOWN } } }
I20260929 10:28:40.599033 28866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:40.595417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40460 (local address 127.20.92.1:46741)
0929 10:28:40.595532 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:40.595535 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:40.595551 (+ 16us) server_negotiation.cc:407] Connection header received
0929 10:28:40.595709 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:40.595711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:40.595755 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:40.595820 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:28:40.596126 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.596638 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.597824 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.598034 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.598110 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:40.598430 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:28:40.598500 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:28:40.598559 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:40.598721 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 10:28:40.598765 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 10:28:40.608222 28927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:40.604721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:46741 (local address 127.20.92.1:41437)
0929 10:28:40.604915 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:40.604934 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:40.605014 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:40.605254 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:40.605257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:40.605266 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:28:40.605373 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:40.605382 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.606361 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.606364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:40.606940 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:40.606948 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.607523 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.607525 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:40.607611 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:40.607637 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:28:40.607932 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:28:40.607936 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:28:40.607987 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 10:28:40.608063 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":18}
I20260929 10:28:40.608237 28866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:40.604883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:41437 (local address 127.20.92.1:46741)
0929 10:28:40.605006 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:40.605010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:40.605026 (+ 16us) server_negotiation.cc:407] Connection header received
0929 10:28:40.605087 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:40.605090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:40.605140 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:40.605254 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:28:40.605497 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.606257 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.607063 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:40.607410 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:40.607510 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:40.607755 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:28:40.607842 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:28:40.607925 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:40.608008 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 10:28:40.608066 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":51}
I20260929 10:28:40.614576 28656 raft_consensus.cc:1261] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 93dc1ff4066c4d21bd37e6241101305c [term 1 FOLLOWER]: Refusing update from remote peer a4679824ad5f48f4a46afdefd2c96a94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:40.614667 28796 raft_consensus.cc:1261] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 3a5bf278a026465bad9f17660385bff0 [term 1 FOLLOWER]: Refusing update from remote peer a4679824ad5f48f4a46afdefd2c96a94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:28:40.614987 28918 consensus_queue.cc:1035] T c45ab7e9a9bc4f9dbed21bc79dd3832c P a4679824ad5f48f4a46afdefd2c96a94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93dc1ff4066c4d21bd37e6241101305c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:28:40.615216 28882 consensus_queue.cc:1035] T c45ab7e9a9bc4f9dbed21bc79dd3832c P a4679824ad5f48f4a46afdefd2c96a94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a5bf278a026465bad9f17660385bff0" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:28:40.649415 28357 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:28:40.649588 28357 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:28:40.651636 28357 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4bcb22f0ca84d1e92a09724ab98d198: Sending DeleteTablet for 3 replicas of tablet c45ab7e9a9bc4f9dbed21bc79dd3832c
I20260929 10:28:40.652128 28497 tablet_service.cc:1514] Processing DeleteTablet for tablet c45ab7e9a9bc4f9dbed21bc79dd3832c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:40 UTC) from {username='oryx', principal='oryx/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:40434
I20260929 10:28:40.652248 28637 tablet_service.cc:1514] Processing DeleteTablet for tablet c45ab7e9a9bc4f9dbed21bc79dd3832c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:40 UTC) from {username='oryx', principal='oryx/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:51392
I20260929 10:28:40.652315 28894 tablet_replica.cc:330] T c45ab7e9a9bc4f9dbed21bc79dd3832c P a4679824ad5f48f4a46afdefd2c96a94: stopping tablet replica
I20260929 10:28:40.652390 28777 tablet_service.cc:1514] Processing DeleteTablet for tablet c45ab7e9a9bc4f9dbed21bc79dd3832c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:28:40 UTC) from {username='oryx', principal='oryx/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:44946
I20260929 10:28:40.652436 28895 tablet_replica.cc:330] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 93dc1ff4066c4d21bd37e6241101305c: stopping tablet replica
I20260929 10:28:40.652431 28894 raft_consensus.cc:2229] T c45ab7e9a9bc4f9dbed21bc79dd3832c P a4679824ad5f48f4a46afdefd2c96a94 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:28:40.652554 28895 raft_consensus.cc:2229] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 93dc1ff4066c4d21bd37e6241101305c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:28:40.652606 28894 raft_consensus.cc:2258] T c45ab7e9a9bc4f9dbed21bc79dd3832c P a4679824ad5f48f4a46afdefd2c96a94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:40.652654 28895 raft_consensus.cc:2258] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 93dc1ff4066c4d21bd37e6241101305c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:40.652827 28896 tablet_replica.cc:330] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 3a5bf278a026465bad9f17660385bff0: stopping tablet replica
I20260929 10:28:40.652959 28896 raft_consensus.cc:2229] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 3a5bf278a026465bad9f17660385bff0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:28:40.653077 28896 raft_consensus.cc:2258] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 3a5bf278a026465bad9f17660385bff0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:28:40.653543 28894 ts_tablet_manager.cc:1905] T c45ab7e9a9bc4f9dbed21bc79dd3832c P a4679824ad5f48f4a46afdefd2c96a94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:40.654006 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 28432
I20260929 10:28:40.654064 28895 ts_tablet_manager.cc:1905] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 93dc1ff4066c4d21bd37e6241101305c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:40.655983 28896 ts_tablet_manager.cc:1905] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 3a5bf278a026465bad9f17660385bff0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:28:40.657234 28895 ts_tablet_manager.cc:1918] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 93dc1ff4066c4d21bd37e6241101305c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:28:40.657316 28895 log.cc:1196] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 93dc1ff4066c4d21bd37e6241101305c: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-1/wal/wals/c45ab7e9a9bc4f9dbed21bc79dd3832c
I20260929 10:28:40.657588 28895 ts_tablet_manager.cc:1939] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 93dc1ff4066c4d21bd37e6241101305c: Deleting consensus metadata
I20260929 10:28:40.658646 28896 ts_tablet_manager.cc:1918] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 3a5bf278a026465bad9f17660385bff0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:28:40.658728 28896 log.cc:1196] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 3a5bf278a026465bad9f17660385bff0: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677679287822-20848-0/minicluster-data/ts-2/wal/wals/c45ab7e9a9bc4f9dbed21bc79dd3832c
I20260929 10:28:40.659332 28896 ts_tablet_manager.cc:1939] T c45ab7e9a9bc4f9dbed21bc79dd3832c P 3a5bf278a026465bad9f17660385bff0: Deleting consensus metadata
I20260929 10:28:40.660207 28346 catalog_manager.cc:4841] TS 3a5bf278a026465bad9f17660385bff0 (127.20.92.3:38451): tablet c45ab7e9a9bc4f9dbed21bc79dd3832c (table test-table [id=bbccc1e669a64eed89eaa66d2123f5a4]) successfully deleted
I20260929 10:28:40.660754 28344 catalog_manager.cc:4841] TS 93dc1ff4066c4d21bd37e6241101305c (127.20.92.2:44115): tablet c45ab7e9a9bc4f9dbed21bc79dd3832c (table test-table [id=bbccc1e669a64eed89eaa66d2123f5a4]) successfully deleted
W20260929 10:28:40.664572 28345 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.92.1:46741 (error 108)
I20260929 10:28:40.664891 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 28572
I20260929 10:28:40.666667 28964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:40.666061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:46741 (local address 127.0.0.1:40470)
0929 10:28:40.666490 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:40.666577 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.1:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":175,"threads_started":1}
W20260929 10:28:40.666810 28345 catalog_manager.cc:4584] TS a4679824ad5f48f4a46afdefd2c96a94 (127.20.92.1:46741): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c45ab7e9a9bc4f9dbed21bc79dd3832c: Network error: Client connection negotiation failed: client connection to 127.20.92.1:46741: connect: Connection refused (error 111)
I20260929 10:28:40.672711 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 28712
I20260929 10:28:40.679559 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 28328
2026-09-29T10:28:40Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3294 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:28:40 dist-test-slave-9cj5 krb5kdc[28967](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:28:40 dist-test-slave-9cj5 krb5kdc[28967](info): set up 2 sockets
Sep 29 10:28:40 dist-test-slave-9cj5 krb5kdc[28967](info): commencing operation
krb5kdc: starting...
W20260929 10:28:41.743820 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:28:41 dist-test-slave-9cj5 krb5kdc[28967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:28:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:28:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
I20260929 10:28:41.890772 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:37157
--webserver_interface=127.20.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:46703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:37157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:42.005385 28980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:42.005764 28980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:42.005889 28980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:42.010142 28980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:28:42.010255 28980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:42.010293 28980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:42.010335 28980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:28:42.010370 28980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:28:42.016105 28980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:46703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:37157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:37157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:42.017253 28980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:42.018321 28980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:28:42.027489 28980 server_base.cc:1034] running on GCE node
W20260929 10:28:42.027444 28985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:42.027441 28986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:42.027444 28988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:42.028451 28980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:42.029696 28980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:42.030941 28980 hybrid_clock.cc:648] HybridClock initialized: now 1790677722030920 us; error 34 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T10:28:42Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1358 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:28:42 dist-test-slave-9cj5 krb5kdc[28993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:28:42 dist-test-slave-9cj5 krb5kdc[28993](info): set up 2 sockets
Sep 29 10:28:42 dist-test-slave-9cj5 krb5kdc[28993](info): commencing operation
krb5kdc: starting...
W20260929 10:28:43.143266 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:28:43 dist-test-slave-9cj5 krb5kdc[28993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:28:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:28:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
I20260929 10:28:43.268859 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:40183
--webserver_interface=127.20.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:44549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:40183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:43.411638 29006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:43.412017 29006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:43.412103 29006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:43.417398 29006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:28:43.417536 29006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:43.417573 29006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:43.417615 29006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:28:43.417685 29006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:28:43.425288 29006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:44549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:40183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:40183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:43.426710 29006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:43.427917 29006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:43.439163 29014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:43.439186 29011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:43.439178 29012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:43.439548 29006 server_base.cc:1034] running on GCE node
I20260929 10:28:43.440235 29006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:43.441516 29006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:43.442893 29006 hybrid_clock.cc:648] HybridClock initialized: now 1790677723442842 us; error 74 us; skew 500 ppm
Sep 29 10:28:43 dist-test-slave-9cj5 krb5kdc[28993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677723, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:43.446756 29006 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:28:43.447854 29006 webserver.cc:460] Webserver started at http://127.20.92.62:44731/ using document root <none> and password file <none>
I20260929 10:28:43.448493 29006 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:43.448590 29006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:43.448832 29006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:43.451042 29006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "10eee167a3f24f24a310b032ff48f125"
format_stamp: "Formatted at 2026-09-29 10:28:43 on dist-test-slave-9cj5"
server_key: "cf24c438cb65cdbeafa58e80265e07e2"
server_key_iv: "d6df438c0fd2fbd5adcba84a492d6ad5"
server_key_version: "encryptionkey@0"
I20260929 10:28:43.451632 29006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "10eee167a3f24f24a310b032ff48f125"
format_stamp: "Formatted at 2026-09-29 10:28:43 on dist-test-slave-9cj5"
server_key: "cf24c438cb65cdbeafa58e80265e07e2"
server_key_iv: "d6df438c0fd2fbd5adcba84a492d6ad5"
server_key_version: "encryptionkey@0"
I20260929 10:28:43.455672 29006 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:28:43.458351 29021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:43.459795 29006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:28:43.459959 29006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "10eee167a3f24f24a310b032ff48f125"
format_stamp: "Formatted at 2026-09-29 10:28:43 on dist-test-slave-9cj5"
server_key: "cf24c438cb65cdbeafa58e80265e07e2"
server_key_iv: "d6df438c0fd2fbd5adcba84a492d6ad5"
server_key_version: "encryptionkey@0"
I20260929 10:28:43.460088 29006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:43.477005 29006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:43.480607 29006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:43.481017 29006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:43.489020 29006 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:40183
I20260929 10:28:43.489032 29073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:40183 every 8 connection(s)
I20260929 10:28:43.490092 29006 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:28:43.493052 29074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:28:43.497332 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 29006
I20260929 10:28:43.497509 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:28:43.497892 20848 external_mini_cluster.cc:1428] Setting key e50eee12e14fe794858fa4aa0c742dc8
I20260929 10:28:43.499876 29074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125: Bootstrap starting.
I20260929 10:28:43.502586 29074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125: Neither blocks nor log segments found. Creating new log.
I20260929 10:28:43.503522 29074 log.cc:824] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:43.506805 29074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125: No bootstrap required, opened a new log
Sep 29 10:28:43 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:43.512148 29074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10eee167a3f24f24a310b032ff48f125" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 40183 } }
I20260929 10:28:43.512491 29074 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:28:43.512537 29074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10eee167a3f24f24a310b032ff48f125, State: Initialized, Role: FOLLOWER
I20260929 10:28:43.513134 29074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10eee167a3f24f24a310b032ff48f125" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 40183 } }
I20260929 10:28:43.513310 29074 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:28:43.513409 29074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:28:43.513523 29074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:28:43.515043 29074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10eee167a3f24f24a310b032ff48f125" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 40183 } }
I20260929 10:28:43.515436 29074 leader_election.cc:302] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10eee167a3f24f24a310b032ff48f125; no voters:
I20260929 10:28:43.515785 29074 leader_election.cc:288] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:28:43.516182 29080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:28:43.516424 29080 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [term 1 LEADER]: Becoming Leader. State: Replica: 10eee167a3f24f24a310b032ff48f125, State: Running, Role: LEADER
I20260929 10:28:43.516769 29077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:43.499350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48812 (local address 127.20.92.62:40183)
0929 10:28:43.499791 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:43.499806 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:43.499843 (+ 37us) server_negotiation.cc:407] Connection header received
0929 10:28:43.500539 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:43.500557 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:43.500891 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:43.501176 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:43.502284 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:43.503816 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:43.504793 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:43.505321 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:43.505507 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:43.508969 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:43.508977 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:43.508999 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:43.509040 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:43.512172 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:43.512953 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:43.512959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:43.512964 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:43.513084 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:43.513458 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:43.513468 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:43.513493 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:43.514065 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:43.514232 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:43.514712 (+ 480us) server_negotiation.cc:299] Negotiation successful
0929 10:28:43.515093 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":134,"threads_started":1}
I20260929 10:28:43.516790 29080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10eee167a3f24f24a310b032ff48f125" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 40183 } }
I20260929 10:28:43.517220 29074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:28:43.518180 29081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10eee167a3f24f24a310b032ff48f125. Latest consensus state: current_term: 1 leader_uuid: "10eee167a3f24f24a310b032ff48f125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10eee167a3f24f24a310b032ff48f125" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 40183 } } }
I20260929 10:28:43.518337 29081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:43.519009 29079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10eee167a3f24f24a310b032ff48f125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10eee167a3f24f24a310b032ff48f125" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 40183 } } }
I20260929 10:28:43.519129 29079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [sys.catalog]: This master's current role is: LEADER
W20260929 10:28:43.522249 29094 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:28:43.522343 29094 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:28:43.522413 29088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:28:43.523160 29088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:28:43.529201 29088 catalog_manager.cc:1347] Generated new cluster ID: fcdc1da34ccc44509863e8ec3ad9f357
I20260929 10:28:43.529294 29088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:28:43.561530 29088 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:28:43.562891 29088 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:28:43.571275 29088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125: Generated new TSK 0
I20260929 10:28:43.572125 29088 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
I20260929 10:28:43.606813 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:0
--local_ip_for_outbound_sockets=127.20.92.1
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:40183
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:44549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:43.723932 29100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:43.724190 29100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:43.724234 29100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:43.728506 29100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:43.728575 29100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:43.728637 29100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:28:43.734692 29100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:44549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:40183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:43.735673 29100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:43.736630 29100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:43.746357 29106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:43.746423 29108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:43.746347 29105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:43.746770 29100 server_base.cc:1034] running on GCE node
I20260929 10:28:43.747197 29100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:43.747815 29100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:43.748991 29100 hybrid_clock.cc:648] HybridClock initialized: now 1790677723748972 us; error 29 us; skew 500 ppm
Sep 29 10:28:43 dist-test-slave-9cj5 krb5kdc[28993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677723, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:43.751724 29100 init.cc:377] Logged in from keytab as kudu/127.20.92.1@KRBTEST.COM (short username kudu)
I20260929 10:28:43.752633 29100 webserver.cc:460] Webserver started at http://127.20.92.1:36263/ using document root <none> and password file <none>
I20260929 10:28:43.753197 29100 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:43.753286 29100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:43.753533 29100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:43.755594 29100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-0/data/instance:
uuid: "6a823b662ef340e983e0f27749333ae7"
format_stamp: "Formatted at 2026-09-29 10:28:43 on dist-test-slave-9cj5"
server_key: "5bf543452ee27340598e42be525ef37f"
server_key_iv: "1d87dd0da7d316303b647b971598bd38"
server_key_version: "encryptionkey@0"
I20260929 10:28:43.756146 29100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance:
uuid: "6a823b662ef340e983e0f27749333ae7"
format_stamp: "Formatted at 2026-09-29 10:28:43 on dist-test-slave-9cj5"
server_key: "5bf543452ee27340598e42be525ef37f"
server_key_iv: "1d87dd0da7d316303b647b971598bd38"
server_key_version: "encryptionkey@0"
I20260929 10:28:43.759918 29100 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 10:28:43.762571 29115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:43.763983 29100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:28:43.764109 29100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "6a823b662ef340e983e0f27749333ae7"
format_stamp: "Formatted at 2026-09-29 10:28:43 on dist-test-slave-9cj5"
server_key: "5bf543452ee27340598e42be525ef37f"
server_key_iv: "1d87dd0da7d316303b647b971598bd38"
server_key_version: "encryptionkey@0"
I20260929 10:28:43.764223 29100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:43.799026 29100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:43.803377 29100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:43.803583 29100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:43.804396 29100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:43.805528 29100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:43.805605 29100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:43.805665 29100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:43.805691 29100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:43.814446 29100 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:37583
I20260929 10:28:43.814471 29228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:37583 every 8 connection(s)
I20260929 10:28:43.815284 29100 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
I20260929 10:28:43.823809 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 29100
I20260929 10:28:43.823956 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance
I20260929 10:28:43.824261 20848 external_mini_cluster.cc:1428] Setting key 71df696f04c8596a73a468947874d955
Sep 29 10:28:43 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677723, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:43.828976 29077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:43.817021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:46469 (local address 127.20.92.62:40183)
0929 10:28:43.817155 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:43.817158 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:43.818072 (+ 914us) server_negotiation.cc:407] Connection header received
0929 10:28:43.819466 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:43.819469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:43.819522 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:43.819627 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:43.820771 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:43.821306 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:43.822492 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:43.822724 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:43.822835 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:43.825398 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:43.825401 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:43.825404 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:43.825429 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:43.826907 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:43.827437 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:43.827442 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:43.827444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:43.827527 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:43.827881 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:43.827884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:43.827885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:43.828067 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:43.828193 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:43.828670 (+ 477us) server_negotiation.cc:299] Negotiation successful
0929 10:28:43.828798 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 10:28:43.830070 29231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:43.817401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.1:46469)
0929 10:28:43.817894 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:43.817935 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:43.819267 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:43.819799 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:43.819805 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:43.820155 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:43.820618 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:43.820631 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:43.821439 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:43.821448 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:43.822367 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:43.822376 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:43.822873 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:43.822878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:43.823040 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:43.823587 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:43.823603 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:43.825239 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:43.827028 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:43.827033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:43.827048 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:43.827310 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:43.827620 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:43.827625 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:43.827627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:43.827775 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:43.828174 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:43.828181 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:43.828429 (+ 248us) client_negotiation.cc:769] Sending connection context
0929 10:28:43.828617 (+ 188us) client_negotiation.cc:240] Negotiation successful
0929 10:28:43.828947 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":108,"threads_started":1}
I20260929 10:28:43.831411 29229 heartbeater.cc:347] Connected to a master server at 127.20.92.62:40183
I20260929 10:28:43.831681 29229 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:43.832227 29229 heartbeater.cc:511] Master 127.20.92.62:40183 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.92.2@KRBTEST.COM; defaulting to no policy
I20260929 10:28:43.833964 29038 ts_manager.cc:194] Registered new tserver with Master: 6a823b662ef340e983e0f27749333ae7 (127.20.92.1:37583)
I20260929 10:28:43.835125 29038 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.1@KRBTEST.COM'} at 127.20.92.1:46469
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
I20260929 10:28:43.852962 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:0
--local_ip_for_outbound_sockets=127.20.92.2
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:40183
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:44549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:43.964435 29234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:43.964709 29234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:43.964781 29234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:43.968856 29234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:43.968930 29234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:43.969022 29234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:28:43.974176 29234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:44549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:40183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:43.975206 29234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:43.976193 29234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:43.985415 29242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:43.985430 29240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:43.985602 29234 server_base.cc:1034] running on GCE node
W20260929 10:28:43.985414 29239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:43.986403 29234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:43.987018 29234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:43.988201 29234 hybrid_clock.cc:648] HybridClock initialized: now 1790677723988172 us; error 45 us; skew 500 ppm
Sep 29 10:28:43 dist-test-slave-9cj5 krb5kdc[28993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677723, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:43.991088 29234 init.cc:377] Logged in from keytab as kudu/127.20.92.2@KRBTEST.COM (short username kudu)
I20260929 10:28:43.991955 29234 webserver.cc:460] Webserver started at http://127.20.92.2:39705/ using document root <none> and password file <none>
I20260929 10:28:43.992511 29234 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:43.992600 29234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:43.992832 29234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:43.994884 29234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-1/data/instance:
uuid: "26c1fe2c27fc424c8c858dcc9acbdb34"
format_stamp: "Formatted at 2026-09-29 10:28:43 on dist-test-slave-9cj5"
server_key: "b153f5bceae836582c3c9170dc74b222"
server_key_iv: "74d73f270e855a705ef88aa9fdc22daa"
server_key_version: "encryptionkey@0"
I20260929 10:28:43.995436 29234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance:
uuid: "26c1fe2c27fc424c8c858dcc9acbdb34"
format_stamp: "Formatted at 2026-09-29 10:28:43 on dist-test-slave-9cj5"
server_key: "b153f5bceae836582c3c9170dc74b222"
server_key_iv: "74d73f270e855a705ef88aa9fdc22daa"
server_key_version: "encryptionkey@0"
I20260929 10:28:43.999158 29234 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:28:44.001538 29249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:44.002774 29234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:28:44.002918 29234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "26c1fe2c27fc424c8c858dcc9acbdb34"
format_stamp: "Formatted at 2026-09-29 10:28:43 on dist-test-slave-9cj5"
server_key: "b153f5bceae836582c3c9170dc74b222"
server_key_iv: "74d73f270e855a705ef88aa9fdc22daa"
server_key_version: "encryptionkey@0"
I20260929 10:28:44.003029 29234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:44.014740 29234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:44.018146 29234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:44.018402 29234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:44.019129 29234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:44.020089 29234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:44.020157 29234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:44.020220 29234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:44.020249 29234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:44.029420 29234 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:38493
I20260929 10:28:44.029439 29362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:38493 every 8 connection(s)
I20260929 10:28:44.030390 29234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
I20260929 10:28:44.039553 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 29234
I20260929 10:28:44.039706 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance
I20260929 10:28:44.040019 20848 external_mini_cluster.cc:1428] Setting key 9b79df96c0c21c720616bb5af65e9808
Sep 29 10:28:44 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677723, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:44.044829 29077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:44.032235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:54397 (local address 127.20.92.62:40183)
0929 10:28:44.032379 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:44.032382 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:44.033225 (+ 843us) server_negotiation.cc:407] Connection header received
0929 10:28:44.034600 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:44.034604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:44.034663 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:44.034765 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:44.035914 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:44.036438 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:44.037790 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:44.038031 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:44.038155 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:44.040825 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:44.040828 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:44.040830 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:44.040852 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:44.042387 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:44.043088 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:44.043093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:44.043096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:44.043203 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:44.043602 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:44.043606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:44.043609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:44.043880 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:44.043999 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:44.044593 (+ 594us) server_negotiation.cc:299] Negotiation successful
0929 10:28:44.044703 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 10:28:44.046190 29365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:44.032614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.2:54397)
0929 10:28:44.033065 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:44.033107 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:44.034380 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:44.034929 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:44.034936 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:44.035290 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:44.035776 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:44.035789 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:44.036566 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:44.036575 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:44.037533 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:44.037542 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:44.038198 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:44.038202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:44.038399 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:44.039016 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:44.039032 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:44.040646 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:44.042506 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:44.042515 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:44.042534 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:44.042946 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:44.043333 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:44.043335 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:44.043337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:44.043480 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:44.044013 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:44.044018 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:44.044328 (+ 310us) client_negotiation.cc:769] Sending connection context
0929 10:28:44.044543 (+ 215us) client_negotiation.cc:240] Negotiation successful
0929 10:28:44.044911 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":109,"threads_started":1}
I20260929 10:28:44.047487 29363 heartbeater.cc:347] Connected to a master server at 127.20.92.62:40183
I20260929 10:28:44.047745 29363 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:44.048265 29363 heartbeater.cc:511] Master 127.20.92.62:40183 requested a full tablet report, sending...
I20260929 10:28:44.049373 29038 ts_manager.cc:194] Registered new tserver with Master: 26c1fe2c27fc424c8c858dcc9acbdb34 (127.20.92.2:38493)
WARNING: no policy specified for kudu/127.20.92.3@KRBTEST.COM; defaulting to no policy
I20260929 10:28:44.050021 29038 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.2@KRBTEST.COM'} at 127.20.92.2:54397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
I20260929 10:28:44.069716 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:0
--local_ip_for_outbound_sockets=127.20.92.3
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:40183
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:44549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:44.178627 29368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:44.178941 29368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:44.179018 29368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:44.183121 29368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:44.183204 29368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:44.183295 29368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:28:44.189674 29368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:44549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:40183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:44.191025 29368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:44.191969 29368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:44.201651 29374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:44.201645 29373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:44.201645 29376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:44.201692 29368 server_base.cc:1034] running on GCE node
I20260929 10:28:44.202365 29368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:44.203035 29368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:44.204258 29368 hybrid_clock.cc:648] HybridClock initialized: now 1790677724204213 us; error 59 us; skew 500 ppm
Sep 29 10:28:44 dist-test-slave-9cj5 krb5kdc[28993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677724, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:44.207368 29368 init.cc:377] Logged in from keytab as kudu/127.20.92.3@KRBTEST.COM (short username kudu)
I20260929 10:28:44.208264 29368 webserver.cc:460] Webserver started at http://127.20.92.3:40101/ using document root <none> and password file <none>
I20260929 10:28:44.208839 29368 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:44.208930 29368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:44.209163 29368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:28:44.211277 29368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-2/data/instance:
uuid: "4deaef4aca3540c0a7ce15fd01955bf5"
format_stamp: "Formatted at 2026-09-29 10:28:44 on dist-test-slave-9cj5"
server_key: "5104a1fabccb312fb9e1703dd491f4b4"
server_key_iv: "47e990f40b793bb54324a10ec01b200f"
server_key_version: "encryptionkey@0"
I20260929 10:28:44.211831 29368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance:
uuid: "4deaef4aca3540c0a7ce15fd01955bf5"
format_stamp: "Formatted at 2026-09-29 10:28:44 on dist-test-slave-9cj5"
server_key: "5104a1fabccb312fb9e1703dd491f4b4"
server_key_iv: "47e990f40b793bb54324a10ec01b200f"
server_key_version: "encryptionkey@0"
I20260929 10:28:44.215612 29368 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 10:28:44.218140 29383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:44.219367 29368 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 10:28:44.219499 29368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "4deaef4aca3540c0a7ce15fd01955bf5"
format_stamp: "Formatted at 2026-09-29 10:28:44 on dist-test-slave-9cj5"
server_key: "5104a1fabccb312fb9e1703dd491f4b4"
server_key_iv: "47e990f40b793bb54324a10ec01b200f"
server_key_version: "encryptionkey@0"
I20260929 10:28:44.219616 29368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:44.242257 29368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:44.245541 29368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:44.245831 29368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:44.246469 29368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:44.247309 29368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:44.247404 29368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:44.247494 29368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:44.247541 29368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:44.256655 29368 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:37495
I20260929 10:28:44.256675 29496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:37495 every 8 connection(s)
I20260929 10:28:44.257536 29368 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:28:44.266506 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 29368
I20260929 10:28:44.266613 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance
Sep 29 10:28:44 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677724, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:28:44.266968 20848 external_mini_cluster.cc:1428] Setting key 7b2e8bd096e11b0593cb5a17febbde9e
I20260929 10:28:44.271312 29077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:44.259300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:37553 (local address 127.20.92.62:40183)
0929 10:28:44.259434 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:44.259440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:44.260150 (+ 710us) server_negotiation.cc:407] Connection header received
0929 10:28:44.261427 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:44.261430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:44.261483 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:44.261596 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:44.262843 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:44.263382 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:44.264602 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:44.264833 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:44.264945 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:44.267496 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:44.267500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:44.267503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:44.267535 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:44.269340 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:44.269848 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:44.269852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:44.269854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:44.269937 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:44.270280 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:44.270283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:44.270285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:44.270463 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:44.270563 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:44.271045 (+ 482us) server_negotiation.cc:299] Negotiation successful
0929 10:28:44.271157 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 10:28:44.272073 29499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:44.259599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.3:37553)
0929 10:28:44.260010 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:44.260049 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:44.261226 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:44.261823 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:44.261831 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:44.262232 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:44.262681 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:44.262696 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:44.263527 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:44.263531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:44.264480 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:44.264490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:44.264984 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:44.264988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:44.265148 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:44.265701 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:44.265721 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:44.267320 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:28:44.269453 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:44.269458 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:44.269472 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:44.269723 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:44.270034 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:28:44.270036 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:28:44.270039 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:28:44.270171 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:28:44.270573 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:28:44.270581 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:28:44.270815 (+ 234us) client_negotiation.cc:769] Sending connection context
0929 10:28:44.270989 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 10:28:44.271310 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":102,"threads_started":1}
I20260929 10:28:44.273137 29497 heartbeater.cc:347] Connected to a master server at 127.20.92.62:40183
I20260929 10:28:44.273355 29497 heartbeater.cc:464] Registering TS with master...
I20260929 10:28:44.273839 29497 heartbeater.cc:511] Master 127.20.92.62:40183 requested a full tablet report, sending...
I20260929 10:28:44.274766 29038 ts_manager.cc:194] Registered new tserver with Master: 4deaef4aca3540c0a7ce15fd01955bf5 (127.20.92.3:37495)
I20260929 10:28:44.275286 29038 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.3@KRBTEST.COM'} at 127.20.92.3:37553
I20260929 10:28:44.280843 20848 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.20.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.1.keytab.
Entry for principal oryx/127.20.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.1.keytab.
WARNING: no policy specified for oryx/127.20.92.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.92.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.2.keytab.
Entry for principal oryx/127.20.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.2.keytab.
WARNING: no policy specified for oryx/127.20.92.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.92.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.3.keytab.
Entry for principal oryx/127.20.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.3.keytab.
I20260929 10:28:44.357575 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 29100
I20260929 10:28:44.364025 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 29234
I20260929 10:28:44.369269 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 29368
I20260929 10:28:44.377990 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 29006
I20260929 10:28:44.384441 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:40183
--webserver_interface=127.20.92.62
--webserver_port=44731
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:44549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:40183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:28:44.494956 29506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:44.495268 29506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:44.495373 29506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:44.499398 29506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:28:44.499495 29506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:44.499536 29506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:44.499571 29506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:28:44.499603 29506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:28:44.504853 29506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:44549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:40183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:40183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=44731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:44.506011 29506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:44.507040 29506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:44.515959 29514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:44.516139 29506 server_base.cc:1034] running on GCE node
W20260929 10:28:44.515969 29511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:44.515964 29512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:44.517050 29506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:44.518496 29506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:44.519702 29506 hybrid_clock.cc:648] HybridClock initialized: now 1790677724519684 us; error 53 us; skew 500 ppm
Sep 29 10:28:44 dist-test-slave-9cj5 krb5kdc[28993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677724, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:44.522614 29506 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:28:44.523452 29506 webserver.cc:460] Webserver started at http://127.20.92.62:44731/ using document root <none> and password file <none>
I20260929 10:28:44.523967 29506 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:44.524040 29506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:44.527973 29506 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 10:28:44.530292 29521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:44.531556 29506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:28:44.531667 29506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "10eee167a3f24f24a310b032ff48f125"
format_stamp: "Formatted at 2026-09-29 10:28:43 on dist-test-slave-9cj5"
server_key: "cf24c438cb65cdbeafa58e80265e07e2"
server_key_iv: "d6df438c0fd2fbd5adcba84a492d6ad5"
server_key_version: "encryptionkey@0"
I20260929 10:28:44.532159 29506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:44.548804 29506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:44.552358 29506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:44.552517 29506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:44.559450 29506 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:40183
I20260929 10:28:44.559465 29573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:40183 every 8 connection(s)
I20260929 10:28:44.560487 29506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:28:44.560992 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 29506
I20260929 10:28:44.561445 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:37583
--local_ip_for_outbound_sockets=127.20.92.1
--tserver_master_addrs=127.20.92.62:40183
--webserver_port=36263
--webserver_interface=127.20.92.1
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:44549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.92.1
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:28:44.565758 29574 sys_catalog.cc:263] Verifying existing consensus state
I20260929 10:28:44.568379 29574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125: Bootstrap starting.
I20260929 10:28:44.591647 29574 log.cc:824] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125: Log is configured to *not* fsync() on all Append() calls
I20260929 10:28:44.600431 29574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:28:44.600898 29574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125: Bootstrap complete.
I20260929 10:28:44.606982 29574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10eee167a3f24f24a310b032ff48f125" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 40183 } }
I20260929 10:28:44.607604 29574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10eee167a3f24f24a310b032ff48f125, State: Initialized, Role: FOLLOWER
I20260929 10:28:44.608234 29574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10eee167a3f24f24a310b032ff48f125" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 40183 } }
I20260929 10:28:44.608412 29574 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:28:44.608486 29574 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:28:44.608587 29574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [term 1 FOLLOWER]: Advancing to term 2
I20260929 10:28:44.611272 29574 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10eee167a3f24f24a310b032ff48f125" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 40183 } }
I20260929 10:28:44.611665 29574 leader_election.cc:302] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10eee167a3f24f24a310b032ff48f125; no voters:
I20260929 10:28:44.612032 29574 leader_election.cc:288] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 10:28:44.612176 29578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 10:28:44.612449 29578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [term 2 LEADER]: Becoming Leader. State: Replica: 10eee167a3f24f24a310b032ff48f125, State: Running, Role: LEADER
I20260929 10:28:44.612798 29578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10eee167a3f24f24a310b032ff48f125" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 40183 } }
I20260929 10:28:44.613325 29574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:28:44.614289 29580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10eee167a3f24f24a310b032ff48f125. Latest consensus state: current_term: 2 leader_uuid: "10eee167a3f24f24a310b032ff48f125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10eee167a3f24f24a310b032ff48f125" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 40183 } } }
I20260929 10:28:44.614257 29579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "10eee167a3f24f24a310b032ff48f125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10eee167a3f24f24a310b032ff48f125" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 40183 } } }
I20260929 10:28:44.614392 29580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:44.614392 29579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125 [sys.catalog]: This master's current role is: LEADER
I20260929 10:28:44.614928 29587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:28:44.618472 29587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:28:44.619256 29587 catalog_manager.cc:1259] Loaded cluster ID: fcdc1da34ccc44509863e8ec3ad9f357
I20260929 10:28:44.619331 29587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:28:44.621148 29587 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:28:44.621706 29587 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 10eee167a3f24f24a310b032ff48f125: Loaded TSK: 0
I20260929 10:28:44.622632 29587 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 10:28:44.706034 29576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:44.706321 29576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:44.706414 29576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:44.710645 29576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:44.710729 29576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:44.710821 29576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:28:44.716753 29576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:44549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.1.keytab
--principal=oryx/127.20.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:37583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=36263
--tserver_master_addrs=127.20.92.62:40183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:44.717810 29576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:44.718798 29576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:44.728103 29603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:44.728097 29600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:44.728150 29601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:44.728888 29576 server_base.cc:1034] running on GCE node
I20260929 10:28:44.729265 29576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:44.729969 29576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:44.731135 29576 hybrid_clock.cc:648] HybridClock initialized: now 1790677724731114 us; error 33 us; skew 500 ppm
Sep 29 10:28:44 dist-test-slave-9cj5 krb5kdc[28993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677724, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:44.734161 29576 init.cc:377] Logged in from keytab as oryx/127.20.92.1@KRBTEST.COM (short username oryx)
I20260929 10:28:44.735148 29576 webserver.cc:460] Webserver started at http://127.20.92.1:36263/ using document root <none> and password file <none>
I20260929 10:28:44.735752 29576 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:44.735828 29576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:44.740074 29576 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:28:44.742369 29610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:44.743754 29576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:28:44.743927 29576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "6a823b662ef340e983e0f27749333ae7"
format_stamp: "Formatted at 2026-09-29 10:28:43 on dist-test-slave-9cj5"
server_key: "5bf543452ee27340598e42be525ef37f"
server_key_iv: "1d87dd0da7d316303b647b971598bd38"
server_key_version: "encryptionkey@0"
I20260929 10:28:44.744508 29576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:44.767834 29576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:44.771085 29576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:44.771311 29576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:44.771921 29576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:44.772732 29576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:44.772804 29576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:44.772872 29576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:44.772924 29576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:44.782097 29576 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:37583
I20260929 10:28:44.782153 29723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:37583 every 8 connection(s)
I20260929 10:28:44.783138 29576 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
I20260929 10:28:44.789422 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 29576
I20260929 10:28:44.789963 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:38493
--local_ip_for_outbound_sockets=127.20.92.2
--tserver_master_addrs=127.20.92.62:40183
--webserver_port=39705
--webserver_interface=127.20.92.2
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:44549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.92.2
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 10:28:44 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:44 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 10:28:44.797446 29726 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:44.785236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:41011 (local address 127.20.92.62:40183)
0929 10:28:44.785669 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:44.785689 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:44.785954 (+ 265us) server_negotiation.cc:407] Connection header received
0929 10:28:44.787442 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:44.787453 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:44.787745 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:44.788018 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:44.789245 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:44.790878 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:44.792269 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:44.792748 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:44.792931 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:44.796703 (+ 3772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.1:41011: BlockingRecv error: recv got EOF from 127.20.92.1:41011 (error 108)
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":132,"threads_started":1}
I20260929 10:28:44.797452 29727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:44.785291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.1:41011)
0929 10:28:44.785790 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:44.785842 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:44.787057 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:44.788186 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:44.788193 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:44.788550 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:44.789036 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:44.789049 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:44.791049 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:44.791052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:44.792129 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:44.792141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:44.792920 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:44.792924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:44.793103 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:44.793785 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:44.793810 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:44.796477 (+ 2667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":153,"threads_started":1}
W20260929 10:28:44.798028 29724 heartbeater.cc:644] Failed to heartbeat to 127.20.92.62:40183 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.92.62:40183: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Sep 29 10:28:44 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:44 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:44.803778 29727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:44.798474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.1:43583)
0929 10:28:44.798591 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:44.798607 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:44.798712 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:44.798958 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:44.798961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:44.799124 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:44.799212 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:44.799217 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:44.799946 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:44.799952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:44.800630 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:44.800640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:44.801110 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:44.801114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:44.801200 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:44.801700 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:44.801712 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:44.803629 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 10:28:44.803857 29726 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:44.798609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:43583 (local address 127.20.92.62:40183)
0929 10:28:44.798747 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:44.798753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:44.798770 (+ 17us) server_negotiation.cc:407] Connection header received
0929 10:28:44.798816 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:44.798818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:44.798857 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:44.798945 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:44.799332 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:44.799839 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:44.800785 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:44.800998 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:44.801108 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:44.803753 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.1:43583: BlockingRecv error: recv got EOF from 127.20.92.1:43583 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 10:28:44 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:44 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:44.810256 29727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:44.804421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.1:41747)
0929 10:28:44.804529 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:44.804546 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:44.804623 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:44.804952 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:44.804954 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:44.805127 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:44.805204 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:44.805209 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:44.806382 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:44.806384 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:44.806948 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:44.806957 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:44.807655 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:44.807658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:44.807723 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:44.808264 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:44.808275 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:44.810152 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 10:28:44.810429 29726 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:44.804545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:41747 (local address 127.20.92.62:40183)
0929 10:28:44.804748 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:44.804752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:44.804765 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:28:44.804802 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:44.804805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:44.804846 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:44.804931 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:44.805309 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:44.806253 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:44.807078 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:44.807453 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:44.807560 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:44.810317 (+ 2757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.1:41747: BlockingRecv error: recv got EOF from 127.20.92.1:41747 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
W20260929 10:28:44.810495 29724 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 10:28:44.908198 29728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:44.908480 29728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:44.908524 29728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:44.912730 29728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:44.912793 29728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:44.912854 29728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:28:44.918666 29728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:44549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.2.keytab
--principal=oryx/127.20.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:38493
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=39705
--tserver_master_addrs=127.20.92.62:40183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:44.919734 29728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:44.920723 29728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:44.930819 29733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:44.930832 29734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:44.930958 29728 server_base.cc:1034] running on GCE node
W20260929 10:28:44.931195 29736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:44.931635 29728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:44.932329 29728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:44.933547 29728 hybrid_clock.cc:648] HybridClock initialized: now 1790677724933501 us; error 58 us; skew 500 ppm
Sep 29 10:28:44 dist-test-slave-9cj5 krb5kdc[28993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677724, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:44.936784 29728 init.cc:377] Logged in from keytab as oryx/127.20.92.2@KRBTEST.COM (short username oryx)
I20260929 10:28:44.937891 29728 webserver.cc:460] Webserver started at http://127.20.92.2:39705/ using document root <none> and password file <none>
I20260929 10:28:44.938431 29728 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:44.938508 29728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:44.942523 29728 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260929 10:28:44.944793 29743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:44.946503 29728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:28:44.946624 29728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "26c1fe2c27fc424c8c858dcc9acbdb34"
format_stamp: "Formatted at 2026-09-29 10:28:43 on dist-test-slave-9cj5"
server_key: "b153f5bceae836582c3c9170dc74b222"
server_key_iv: "74d73f270e855a705ef88aa9fdc22daa"
server_key_version: "encryptionkey@0"
I20260929 10:28:44.947183 29728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:44.970562 29728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:44.974028 29728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:44.974188 29728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:44.974706 29728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:44.975540 29728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:44.975592 29728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:44.975631 29728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:44.975646 29728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:44.984664 29728 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:38493
I20260929 10:28:44.984689 29856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:38493 every 8 connection(s)
I20260929 10:28:44.985679 29728 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
I20260929 10:28:44.988300 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 29728
I20260929 10:28:44.988744 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:37495
--local_ip_for_outbound_sockets=127.20.92.3
--tserver_master_addrs=127.20.92.62:40183
--webserver_port=40101
--webserver_interface=127.20.92.3
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:44549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.92.3
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 10:28:44 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:44 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 10:28:45.006191 29726 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:44.988096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:48607 (local address 127.20.92.62:40183)
0929 10:28:44.988270 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:44.988275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:44.989301 (+ 1026us) server_negotiation.cc:407] Connection header received
0929 10:28:44.991007 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:44.991013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:44.991089 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:44.991207 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:44.992892 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:44.993432 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:44.995418 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:44.995644 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:44.995741 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:45.006073 (+ 10332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.2:48607: BlockingRecv error: recv got EOF from 127.20.92.2:48607 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 10:28:45.007004 29859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:44.988563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.2:48607)
0929 10:28:44.989103 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:44.989167 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:44.990739 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:44.991382 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:44.991389 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:44.991987 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:44.992701 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:44.992723 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:44.993570 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:44.993577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:44.995255 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:44.995269 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:44.995831 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:44.995839 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:44.996092 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:45.002692 (+ 6600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:45.002720 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:45.005921 (+ 3201us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":123,"threads_started":1}
W20260929 10:28:45.007622 29857 heartbeater.cc:644] Failed to heartbeat to 127.20.92.62:40183 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.92.62:40183: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Sep 29 10:28:45 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:45 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:45.014933 29859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:45.008173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.2:59091)
0929 10:28:45.008311 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:45.008328 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:45.008426 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:45.008694 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:45.008696 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:45.008887 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:45.009007 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:45.009014 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:45.010122 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:45.010125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:45.010680 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:45.010689 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:45.011357 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:45.011360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:45.011461 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:45.012152 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:45.012166 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:45.014578 (+ 2412us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 10:28:45.014969 29726 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:45.008280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:59091 (local address 127.20.92.62:40183)
0929 10:28:45.008418 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:45.008423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:45.008439 (+ 16us) server_negotiation.cc:407] Connection header received
0929 10:28:45.008536 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:45.008539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:45.008586 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:45.008663 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:45.009130 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:45.009992 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:45.010864 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:45.011213 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:45.011321 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:45.014852 (+ 3531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.2:59091: BlockingRecv error: recv got EOF from 127.20.92.2:59091 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 10:28:45 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:45 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:45.021288 29859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:45.015543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.2:34711)
0929 10:28:45.015671 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:45.015688 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:45.015773 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:45.016004 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:45.016007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:45.016172 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:45.016263 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:45.016269 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:45.017104 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:45.017107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:45.017917 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:45.017931 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:45.018389 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:45.018393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:45.018470 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:45.019015 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:45.019030 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:45.021120 (+ 2090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 10:28:45.021356 29726 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:45.015644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:34711 (local address 127.20.92.62:40183)
0929 10:28:45.015758 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:45.015761 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:45.015774 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:28:45.015861 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:45.015863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:45.015899 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:45.015964 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:45.016405 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:45.016987 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:45.018048 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:45.018268 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:45.018364 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:45.021244 (+ 2880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.2:34711: BlockingRecv error: recv got EOF from 127.20.92.2:34711 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 10:28:45.021530 29857 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 10:28:45.109764 29860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:28:45.110076 29860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:28:45.110152 29860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:28:45.114444 29860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:28:45.114527 29860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:28:45.114629 29860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:28:45.120211 29860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:44549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/krb5kdc/oryx_127.20.92.3.keytab
--principal=oryx/127.20.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:37495
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=40101
--tserver_master_addrs=127.20.92.62:40183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:28:45.121287 29860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:28:45.122371 29860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:28:45.132362 29866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:28:45.132400 29865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:45.132460 29860 server_base.cc:1034] running on GCE node
W20260929 10:28:45.132587 29868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:28:45.133064 29860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:28:45.133827 29860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:28:45.135026 29860 hybrid_clock.cc:648] HybridClock initialized: now 1790677725135000 us; error 41 us; skew 500 ppm
Sep 29 10:28:45 dist-test-slave-9cj5 krb5kdc[28993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677725, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:28:45.138185 29860 init.cc:377] Logged in from keytab as oryx/127.20.92.3@KRBTEST.COM (short username oryx)
I20260929 10:28:45.139123 29860 webserver.cc:460] Webserver started at http://127.20.92.3:40101/ using document root <none> and password file <none>
I20260929 10:28:45.139722 29860 fs_manager.cc:359] Metadata directory not provided
I20260929 10:28:45.139806 29860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:28:45.144451 29860 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 10:28:45.147181 29875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:45.148795 29860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:28:45.148936 29860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "4deaef4aca3540c0a7ce15fd01955bf5"
format_stamp: "Formatted at 2026-09-29 10:28:44 on dist-test-slave-9cj5"
server_key: "5104a1fabccb312fb9e1703dd491f4b4"
server_key_iv: "47e990f40b793bb54324a10ec01b200f"
server_key_version: "encryptionkey@0"
I20260929 10:28:45.149502 29860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:28:45.173461 29860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:28:45.177806 29860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:28:45.178064 29860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:28:45.178723 29860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:28:45.179607 29860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:28:45.179684 29860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:45.179891 29860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:28:45.179972 29860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:28:45.194693 29860 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:37495
I20260929 10:28:45.194710 29988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:37495 every 8 connection(s)
I20260929 10:28:45.196002 29860 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:28:45.197288 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 29860
Sep 29 10:28:45 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:45 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 10:28:45.210528 29726 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:45.198427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:38645 (local address 127.20.92.62:40183)
0929 10:28:45.198576 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:45.198580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:45.200058 (+ 1478us) server_negotiation.cc:407] Connection header received
0929 10:28:45.201340 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:45.201343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:45.201389 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:45.201472 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:45.202880 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:45.203589 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:45.205357 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:45.205739 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:45.205894 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:45.210392 (+ 4498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.3:38645: BlockingRecv error: recv got EOF from 127.20.92.3:38645 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 10:28:45.210911 29992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:45.198809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48822 (local address 127.20.92.62:40183)
0929 10:28:45.199073 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:45.199077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:45.199091 (+ 14us) server_negotiation.cc:407] Connection header received
0929 10:28:45.199136 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:45.199138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:45.199179 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:45.199268 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:45.200275 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:45.201184 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:45.202560 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:45.202969 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:45.203060 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:45.204845 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:28:45.204853 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:28:45.204872 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:28:45.204907 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:28:45.207824 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:45.208769 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:45.208776 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:45.208790 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:45.208931 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:28:45.209263 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:28:45.209272 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:28:45.209273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:28:45.209710 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:28:45.209975 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 10:28:45.210475 (+ 500us) server_negotiation.cc:299] Negotiation successful
0929 10:28:45.210769 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":119,"threads_started":1}
I20260929 10:28:45.211468 29993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:45.199047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.3:38645)
0929 10:28:45.199882 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:45.199939 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:45.201065 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:45.201660 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:45.201667 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:45.202081 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:45.202710 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:45.202731 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:45.203749 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:45.203753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:45.205189 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:45.205205 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:45.205989 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:45.205996 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:45.206364 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:45.206976 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:45.206996 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:45.210274 (+ 3278us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":178,"threads_started":1}
W20260929 10:28:45.212826 29989 heartbeater.cc:644] Failed to heartbeat to 127.20.92.62:40183 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.92.62:40183: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Sep 29 10:28:45 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:45 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:45.221056 29993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:45.214236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.3:39967)
0929 10:28:45.214381 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:45.214399 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:45.214498 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:45.215014 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:45.215016 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:45.215218 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:45.215329 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:45.215335 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:45.216198 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:45.216201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:45.216808 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:45.216819 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:45.217564 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:45.217567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:45.217933 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:45.218446 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:45.218465 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:45.220936 (+ 2471us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260929 10:28:45.221403 29992 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:45.214558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:39967 (local address 127.20.92.62:40183)
0929 10:28:45.214719 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:45.214722 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:45.214735 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:28:45.214768 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:45.214770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:45.214874 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:45.214965 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:45.215464 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:45.216060 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:45.217121 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:45.217379 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:45.217472 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:45.221171 (+ 3699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.3:39967: BlockingRecv error: recv got EOF from 127.20.92.3:39967 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 29 10:28:45 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:45 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:45.227432 29993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:45.221881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.3:58315)
0929 10:28:45.222059 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:45.222080 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:45.222154 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:45.222538 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:45.222540 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:45.222708 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:45.222787 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:45.222800 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:45.223536 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:45.223541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:45.224081 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:45.224089 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:45.224551 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:45.224554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:45.224631 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:45.225290 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:45.225303 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:45.227277 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":85}
W20260929 10:28:45.227553 29992 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:45.222136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:58315 (local address 127.20.92.62:40183)
0929 10:28:45.222339 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:45.222342 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:45.222353 (+ 11us) server_negotiation.cc:407] Connection header received
0929 10:28:45.222387 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:45.222389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:45.222424 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:45.222520 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:45.222913 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:45.223421 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:45.224207 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:45.224431 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:45.224547 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:45.227442 (+ 2895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.3:58315: BlockingRecv error: recv got EOF from 127.20.92.3:58315 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
W20260929 10:28:45.227690 29989 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 10:28:45 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:45 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:45.816776 29994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:45.811090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.1:34279)
0929 10:28:45.811345 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:45.811360 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:45.811463 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:45.811708 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:45.811711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:45.811881 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:45.811997 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:45.812002 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:45.812744 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:45.812746 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:45.813328 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:45.813335 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:45.813850 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:45.813853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:45.813985 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:45.814569 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:45.814588 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:45.816650 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":97,"threads_started":1}
W20260929 10:28:45.816931 29995 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:45.811179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:34279 (local address 127.20.92.62:40183)
0929 10:28:45.811410 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:45.811413 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:45.811426 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:28:45.811544 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:45.811546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:45.811586 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:45.811667 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:45.812107 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:45.812641 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:45.813436 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:45.813678 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:45.813818 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:45.816788 (+ 2970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.1:34279: BlockingRecv error: recv got EOF from 127.20.92.1:34279 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":80,"threads_started":1}
Sep 29 10:28:46 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:46 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:46.028237 29996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:46.022327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.2:46849)
0929 10:28:46.022588 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:46.022603 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:46.022778 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:46.023068 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:46.023071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:46.023287 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:46.023378 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:46.023384 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:46.024196 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:46.024202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:46.024802 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:46.024811 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:46.025305 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:46.025308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:46.025393 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:46.026056 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:46.026069 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:46.028049 (+ 1980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":96,"threads_started":1}
W20260929 10:28:46.028359 29995 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:46.022426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:46849 (local address 127.20.92.62:40183)
0929 10:28:46.022589 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:46.022595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:46.022721 (+ 126us) server_negotiation.cc:407] Connection header received
0929 10:28:46.022901 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:46.022904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:46.022959 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:46.023044 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:46.023508 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:46.024072 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:46.024928 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:46.025182 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:46.025300 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:46.028192 (+ 2892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.2:46849: BlockingRecv error: recv got EOF from 127.20.92.2:46849 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 10:28:46 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:46 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:46.234339 29997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:46.228367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.3:37529)
0929 10:28:46.228669 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:46.228683 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:46.228799 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:46.229084 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:46.229087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:46.229266 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:46.229363 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:46.229368 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:46.230267 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:46.230270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:46.230869 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:46.230879 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:46.231313 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:46.231316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:46.231379 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:46.232004 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:46.232016 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:46.234167 (+ 2151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":86,"threads_started":1}
W20260929 10:28:46.234550 29995 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:46.228537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:37529 (local address 127.20.92.62:40183)
0929 10:28:46.228692 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:46.228696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:46.228747 (+ 51us) server_negotiation.cc:407] Connection header received
0929 10:28:46.228890 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:46.228892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:46.228939 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:46.229012 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:46.229475 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:46.230062 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:46.230988 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:46.231198 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:46.231279 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:46.234400 (+ 3121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.3:37529: BlockingRecv error: recv got EOF from 127.20.92.3:37529 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 10:28:46 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:46 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:46.823873 29998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:46.817584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.1:35305)
0929 10:28:46.817908 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:46.817923 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:46.818019 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:46.818388 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:46.818391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:46.818573 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:46.818672 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:46.818684 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:46.819598 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:46.819601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:46.820181 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:46.820191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:46.820779 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:46.820783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:46.820849 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:46.821422 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:46.821436 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:46.823733 (+ 2297us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":112,"threads_started":1}
W20260929 10:28:46.824085 29999 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:46.817861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:35305 (local address 127.20.92.62:40183)
0929 10:28:46.818104 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:46.818110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:46.818127 (+ 17us) server_negotiation.cc:407] Connection header received
0929 10:28:46.818167 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:46.818169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:46.818213 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:46.818305 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:46.818843 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:46.819462 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:46.820340 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:46.820598 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:46.820750 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:46.823881 (+ 3131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.1:35305: BlockingRecv error: recv got EOF from 127.20.92.1:35305 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":88,"threads_started":1}
Sep 29 10:28:47 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:47 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:47.035205 30000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:47.029167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.2:46125)
0929 10:28:47.029434 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:47.029461 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:47.029561 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:47.029901 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:47.029903 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:47.030071 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:47.030164 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:47.030169 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:47.030966 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:47.030969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:47.031598 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:47.031607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:47.032185 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:47.032188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:47.032255 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:47.032769 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:47.032781 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:47.034994 (+ 2213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":75,"threads_started":1}
W20260929 10:28:47.035279 29999 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:47.029289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:46125 (local address 127.20.92.62:40183)
0929 10:28:47.029459 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:47.029469 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:47.029535 (+ 66us) server_negotiation.cc:407] Connection header received
0929 10:28:47.029705 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:47.029708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:47.029785 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:47.029866 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:47.030274 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:47.030840 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:47.031784 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:47.032050 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:47.032140 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:47.035155 (+ 3015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.2:46125: BlockingRecv error: recv got EOF from 127.20.92.2:46125 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 29 10:28:47 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:47 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:47.241570 30001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:47.235508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.3:48793)
0929 10:28:47.235812 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:47.235826 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:47.235930 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:47.236260 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:47.236263 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:47.236459 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:47.236572 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:47.236579 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:47.237486 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:47.237489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:47.238165 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:47.238175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:47.238801 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:47.238804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:47.238880 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:47.239407 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:47.239420 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:47.241422 (+ 2002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":91,"threads_started":1}
W20260929 10:28:47.241712 29999 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:47.235599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:48793 (local address 127.20.92.62:40183)
0929 10:28:47.235749 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:47.235755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:47.235903 (+ 148us) server_negotiation.cc:407] Connection header received
0929 10:28:47.236046 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:47.236050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:47.236115 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:47.236217 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:47.236755 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:47.237339 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:47.238327 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:47.238580 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:47.238685 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:47.241577 (+ 2892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.3:48793: BlockingRecv error: recv got EOF from 127.20.92.3:48793 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 10:28:47 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:47 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:47.830559 30002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:47.824742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.1:41571)
0929 10:28:47.825054 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:47.825067 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:47.825184 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:47.825547 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:47.825550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:47.825820 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:47.825991 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:47.825999 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:47.826921 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:47.826924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:47.827520 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:47.827528 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:47.827962 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:47.827965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:47.828024 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:47.828517 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:47.828528 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:47.830389 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":134,"threads_started":1}
W20260929 10:28:47.830624 30003 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:47.824882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:41571 (local address 127.20.92.62:40183)
0929 10:28:47.825181 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:47.825187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:47.825205 (+ 18us) server_negotiation.cc:407] Connection header received
0929 10:28:47.825313 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:47.825317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:47.825376 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:47.825468 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:47.826107 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:47.826803 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:47.827627 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:47.827865 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:47.827942 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:47.830504 (+ 2562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.1:41571: BlockingRecv error: recv got EOF from 127.20.92.1:41571 (error 108)
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":116,"threads_started":1}
Sep 29 10:28:48 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:48 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:48.043684 30004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:48.036126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.2:58091)
0929 10:28:48.036503 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:48.036521 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:48.036657 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:48.036966 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:48.036971 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:48.037225 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:48.037365 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:48.037374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:48.038582 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:48.038588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:48.039652 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:48.039667 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:48.040513 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:48.040517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:48.040608 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:48.041181 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:48.041195 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:48.043547 (+ 2352us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":86,"threads_started":1}
W20260929 10:28:48.044124 30003 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:48.036263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:58091 (local address 127.20.92.62:40183)
0929 10:28:48.036454 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:48.036459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:48.036608 (+ 149us) server_negotiation.cc:407] Connection header received
0929 10:28:48.036785 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:48.036787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:48.036833 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:48.036916 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:48.037521 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:48.038447 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:48.039836 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:48.040254 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:48.040386 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:48.043961 (+ 3575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.2:58091: BlockingRecv error: recv got EOF from 127.20.92.2:58091 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 29 10:28:48 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:48 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:48.248270 30005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:48.242438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.3:53389)
0929 10:28:48.242742 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:48.242755 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:48.242908 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:48.243197 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:48.243200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:48.243417 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:48.243503 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:48.243509 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:48.244281 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:48.244286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:48.244876 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:48.244884 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:48.245343 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:48.245346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:48.245447 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:48.246156 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:48.246167 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:48.248079 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":105,"threads_started":1}
W20260929 10:28:48.248304 30003 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:48.242505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:53389 (local address 127.20.92.62:40183)
0929 10:28:48.242632 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:48.242638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:48.242854 (+ 216us) server_negotiation.cc:407] Connection header received
0929 10:28:48.243018 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:48.243022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:48.243074 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:48.243177 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:48.243623 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:48.244170 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:48.244998 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:48.245221 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:48.245333 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:48.248202 (+ 2869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.3:53389: BlockingRecv error: recv got EOF from 127.20.92.3:53389 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 29 10:28:48 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:48 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:48.837008 30006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:48.831561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.1:50469)
0929 10:28:48.831837 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:48.831855 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:48.831993 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:48.832384 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:48.832390 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:48.832635 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:48.832731 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:48.832736 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:48.833474 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:48.833476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:48.834041 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:48.834048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:48.834463 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:48.834465 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:48.834523 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:48.835027 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:48.835038 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:48.836885 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":101,"threads_started":1}
W20260929 10:28:48.837176 30007 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:48.831624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:50469 (local address 127.20.92.62:40183)
0929 10:28:48.831913 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:48.831917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:48.831942 (+ 25us) server_negotiation.cc:407] Connection header received
0929 10:28:48.832144 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:48.832148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:48.832227 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:48.832324 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:48.832840 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:48.833382 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:48.834142 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:48.834369 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:48.834447 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:48.837030 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.1:50469: BlockingRecv error: recv got EOF from 127.20.92.1:50469 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":89,"threads_started":1}
Sep 29 10:28:49 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:49 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 10:28:49.050385 30007 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:49.044700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:60809 (local address 127.20.92.62:40183)
0929 10:28:49.044864 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:49.044868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:49.044959 (+ 91us) server_negotiation.cc:407] Connection header received
0929 10:28:49.045050 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:49.045052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:49.045096 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:49.045200 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:49.045723 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:49.046325 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:49.047160 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:49.047401 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:49.047511 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:49.050280 (+ 2769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.2:60809: BlockingRecv error: recv got EOF from 127.20.92.2:60809 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 10:28:49.050688 30008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:49.044577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.2:60809)
0929 10:28:49.044864 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:49.044877 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:49.044974 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:49.045257 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:49.045260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:49.045492 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:49.045592 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:49.045599 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:49.046466 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:49.046471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:49.047031 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:49.047039 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:49.047515 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:49.047519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:49.047604 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:49.048225 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:49.048238 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:49.050547 (+ 2309us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":101,"threads_started":1}
Sep 29 10:28:49 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:49 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:49.254880 30009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:49.249036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.3:43959)
0929 10:28:49.249353 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:49.249368 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:49.249503 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:49.249837 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:49.249839 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:49.250007 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:49.250104 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:49.250110 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:49.250907 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:49.250909 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:49.251585 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:49.251597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:49.252023 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:49.252026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:49.252120 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:49.252671 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:49.252684 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:49.254761 (+ 2077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":96,"threads_started":1}
W20260929 10:28:49.255127 30007 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:49.249143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:43959 (local address 127.20.92.62:40183)
0929 10:28:49.249323 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:49.249328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:49.249447 (+ 119us) server_negotiation.cc:407] Connection header received
0929 10:28:49.249646 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:49.249648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:49.249697 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:49.249777 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:49.250200 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:49.250785 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:49.251702 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:49.251928 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:49.252004 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:49.255003 (+ 2999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.3:43959: BlockingRecv error: recv got EOF from 127.20.92.3:43959 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 29 10:28:49 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:49 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:49.844018 30010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:49.837889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.1:57273)
0929 10:28:49.838149 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:49.838162 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:49.838257 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:49.838495 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:49.838497 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:49.838672 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:49.838810 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:49.838815 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:49.839647 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:49.839650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:49.840249 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:49.840258 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:49.840808 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:49.840811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:49.840884 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:49.841450 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:49.841463 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:49.843831 (+ 2368us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":91,"threads_started":1}
W20260929 10:28:49.844091 30011 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:49.837899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:57273 (local address 127.20.92.62:40183)
0929 10:28:49.838146 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:49.838149 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:49.838243 (+ 94us) server_negotiation.cc:407] Connection header received
0929 10:28:49.838341 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:49.838343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:49.838385 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:49.838462 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:49.838946 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:49.839506 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:49.840373 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:49.840660 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:49.840771 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:49.843960 (+ 3189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.1:57273: BlockingRecv error: recv got EOF from 127.20.92.1:57273 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":86,"threads_started":1}
Sep 29 10:28:50 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:50 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:50.061889 30012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:50.051552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.2:52755)
0929 10:28:50.051807 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:50.051828 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:50.051975 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:50.052257 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:50.052260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:50.052475 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:50.052562 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:50.052568 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:50.053420 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:50.053425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:50.054025 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:50.054035 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:50.054510 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:50.054513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:50.054600 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:50.059660 (+ 5060us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:50.059673 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:50.061695 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":91,"threads_started":1}
W20260929 10:28:50.062057 30011 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:50.051805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:52755 (local address 127.20.92.62:40183)
0929 10:28:50.051967 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:50.051972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:50.051987 (+ 15us) server_negotiation.cc:407] Connection header received
0929 10:28:50.052066 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:50.052077 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:50.052140 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:50.052238 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:50.052687 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:50.053290 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:50.054155 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:50.054390 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:50.054504 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:50.061904 (+ 7400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.2:52755: BlockingRecv error: recv got EOF from 127.20.92.2:52755 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 10:28:50 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:50 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:50.261484 30013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:50.255796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.3:34619)
0929 10:28:50.256115 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:50.256132 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:50.256289 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:50.256606 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:50.256608 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:50.256834 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:50.256923 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:50.256929 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:50.257690 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:50.257694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:50.258294 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:50.258302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:50.258765 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:50.258769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:50.258852 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:50.259442 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:50.259455 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:50.261314 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":85,"threads_started":1}
W20260929 10:28:50.261565 30011 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:50.255931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:34619 (local address 127.20.92.62:40183)
0929 10:28:50.256097 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:50.256102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:50.256234 (+ 132us) server_negotiation.cc:407] Connection header received
0929 10:28:50.256437 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:50.256439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:50.256486 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:50.256579 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:50.257045 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:50.257579 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:50.258416 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:50.258645 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:50.258764 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:50.261450 (+ 2686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.3:34619: BlockingRecv error: recv got EOF from 127.20.92.3:34619 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 29 10:28:50 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:50 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:50.850538 30014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:50.844890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.1:59869)
0929 10:28:50.845193 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:50.845207 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:50.845296 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:50.845542 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:50.845544 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:50.845717 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:50.845864 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:50.845870 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:50.846590 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:50.846593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:50.847142 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:50.847150 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:50.847611 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:50.847614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:50.847674 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:50.848179 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:50.848190 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:50.850368 (+ 2178us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":85,"threads_started":1}
W20260929 10:28:50.850618 30015 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:50.845008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:59869 (local address 127.20.92.62:40183)
0929 10:28:50.845254 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:50.845257 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:50.845272 (+ 15us) server_negotiation.cc:407] Connection header received
0929 10:28:50.845389 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:50.845391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:50.845432 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:50.845509 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:50.845960 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:50.846490 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:50.847259 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:50.847504 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:50.847593 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:50.850484 (+ 2891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.1:59869: BlockingRecv error: recv got EOF from 127.20.92.1:59869 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":76,"threads_started":1}
Sep 29 10:28:51 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:51 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:51.067767 30016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:51.062594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.2:50547)
0929 10:28:51.062828 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:51.062845 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:51.062948 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:51.063199 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:51.063201 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:51.063387 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:51.063479 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:51.063484 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:51.064174 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:51.064176 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:51.064728 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:51.064735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:51.065121 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:51.065123 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:51.065181 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:51.065659 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:51.065671 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:51.067652 (+ 1981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":82,"threads_started":1}
W20260929 10:28:51.067912 30015 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:51.062632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:50547 (local address 127.20.92.62:40183)
0929 10:28:51.062772 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:51.062776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:51.062923 (+ 147us) server_negotiation.cc:407] Connection header received
0929 10:28:51.063055 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:51.063057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:51.063102 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:51.063171 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:51.063575 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:51.064089 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:51.064824 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:51.065035 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:51.065107 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:51.067786 (+ 2679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.2:50547: BlockingRecv error: recv got EOF from 127.20.92.2:50547 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 10:28:51 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:51 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:51.267741 30017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:51.262328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.3:36925)
0929 10:28:51.262695 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:51.262707 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:51.262813 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:51.263041 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:51.263043 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:51.263220 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:51.263325 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:51.263330 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:51.264049 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:51.264050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:51.264595 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:51.264602 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:51.265008 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:51.265011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:51.265078 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:51.265579 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:51.265591 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:51.267612 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":206,"threads_started":1}
W20260929 10:28:51.267875 30015 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:51.262517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:36925 (local address 127.20.92.62:40183)
0929 10:28:51.262646 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:51.262649 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:51.262776 (+ 127us) server_negotiation.cc:407] Connection header received
0929 10:28:51.262890 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:51.262892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:51.262939 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:51.263009 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:51.263417 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:51.263959 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:51.264685 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:51.264911 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:51.264985 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:51.267755 (+ 2770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.3:36925: BlockingRecv error: recv got EOF from 127.20.92.3:36925 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 10:28:51 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:51 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:51.857378 30018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:51.851359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.1:38991)
0929 10:28:51.851774 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:51.851790 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:51.851937 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:51.852219 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:51.852222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:51.852445 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:51.852534 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:51.852540 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:51.853395 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:51.853399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:51.854005 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:51.854014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:51.854512 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:51.854515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:51.854602 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:51.855125 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:51.855137 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:51.857188 (+ 2051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":88,"threads_started":1}
W20260929 10:28:51.857453 30019 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:51.851476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:38991 (local address 127.20.92.62:40183)
0929 10:28:51.851774 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:51.851780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:51.851872 (+ 92us) server_negotiation.cc:407] Connection header received
0929 10:28:51.852036 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:51.852039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:51.852090 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:51.852186 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:51.852649 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:51.853286 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:51.854163 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:51.854396 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:51.854506 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:51.857327 (+ 2821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.1:38991: BlockingRecv error: recv got EOF from 127.20.92.1:38991 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":82,"threads_started":1}
Sep 29 10:28:52 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:52 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:52.075429 30020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:52.068964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.2:54881)
0929 10:28:52.069286 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:52.069300 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:52.069550 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:52.069994 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:52.069997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:52.070189 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:52.070316 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:52.070323 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:52.071166 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:52.071170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:52.071753 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:52.071762 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:52.072334 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:52.072337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:52.072407 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:52.072964 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:52.072977 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:52.075264 (+ 2287us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":91,"threads_started":1}
W20260929 10:28:52.075553 30019 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:52.069066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:54881 (local address 127.20.92.62:40183)
0929 10:28:52.069250 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:52.069254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:52.069481 (+ 227us) server_negotiation.cc:407] Connection header received
0929 10:28:52.069689 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:52.069691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:52.069847 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:52.069927 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:52.070437 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:52.071043 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:52.071909 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:52.072164 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:52.072276 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:52.075423 (+ 3147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.2:54881: BlockingRecv error: recv got EOF from 127.20.92.2:54881 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 29 10:28:52 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:52 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:52.275076 30021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:52.268713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.3:39195)
0929 10:28:52.269121 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:52.269158 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:52.269339 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:52.269880 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:52.269884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:52.270101 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:52.270254 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:52.270263 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:52.271003 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:52.271009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:52.271562 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:52.271570 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:52.272020 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:52.272023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:52.272115 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:52.272639 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:52.272650 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:52.274904 (+ 2254us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":98,"threads_started":1}
W20260929 10:28:52.275182 30019 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:52.268875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:39195 (local address 127.20.92.62:40183)
0929 10:28:52.269075 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:52.269080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:52.269283 (+ 203us) server_negotiation.cc:407] Connection header received
0929 10:28:52.269577 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:52.269581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:52.269660 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:52.269875 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:52.270363 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:52.270892 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:52.271683 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:52.271909 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:52.272018 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:52.275058 (+ 3040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.3:39195: BlockingRecv error: recv got EOF from 127.20.92.3:39195 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 29 10:28:52 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:52 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:52.864424 30022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:52.858538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.1:37235)
0929 10:28:52.858806 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:52.858826 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:52.858937 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:52.859176 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:52.859179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:52.859388 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:52.859490 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:52.859496 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:52.860295 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:52.860298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:52.860940 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:52.860949 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:52.861404 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:52.861407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:52.861523 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:52.862117 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:52.862135 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:52.864225 (+ 2090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":93,"threads_started":1}
W20260929 10:28:52.864538 30023 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:52.858683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:37235 (local address 127.20.92.62:40183)
0929 10:28:52.858930 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:52.858933 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:52.858947 (+ 14us) server_negotiation.cc:407] Connection header received
0929 10:28:52.859013 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:52.859015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:52.859055 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:52.859135 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:52.859599 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:52.860155 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:52.861061 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:52.861298 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:52.861384 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:52.864393 (+ 3009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.1:37235: BlockingRecv error: recv got EOF from 127.20.92.1:37235 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":78,"threads_started":1}
Sep 29 10:28:53 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:53 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:53.082876 30024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:53.076416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.2:40045)
0929 10:28:53.076724 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:53.076744 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:53.076878 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:53.077182 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:53.077185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:53.077390 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:53.077501 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:53.077507 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:53.078976 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:53.078979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:53.079779 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:53.079790 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:53.080242 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:53.080244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:53.080303 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:53.080782 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:53.080795 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:53.082696 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":131,"threads_started":1}
W20260929 10:28:53.082934 30023 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:53.076556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:40045 (local address 127.20.92.62:40183)
0929 10:28:53.076717 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:53.076721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:53.076842 (+ 121us) server_negotiation.cc:407] Connection header received
0929 10:28:53.076970 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:53.076973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:53.077020 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:53.077109 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:53.077656 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:53.078797 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:53.079904 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:53.080130 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:53.080213 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:53.082825 (+ 2612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.2:40045: BlockingRecv error: recv got EOF from 127.20.92.2:40045 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 10:28:53 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:53 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:53.281635 30025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:53.276008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.3:42631)
0929 10:28:53.276270 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:53.276290 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:53.276399 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:53.276636 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:53.276638 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:53.276865 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:53.276961 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:53.276968 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:53.278000 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:53.278003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:53.278562 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:53.278570 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:53.279101 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:53.279105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:53.279176 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:53.279725 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:53.279737 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:53.281508 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":85,"threads_started":1}
W20260929 10:28:53.281898 30023 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:53.276108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:42631 (local address 127.20.92.62:40183)
0929 10:28:53.276270 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:53.276274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:53.276396 (+ 122us) server_negotiation.cc:407] Connection header received
0929 10:28:53.276467 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:53.276469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:53.276523 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:53.276600 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:53.277102 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:53.277697 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:53.278685 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:53.278956 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:53.279051 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:53.281687 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.3:42631: BlockingRecv error: recv got EOF from 127.20.92.3:42631 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 10:28:53 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:53 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:53.871438 30026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:53.865584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.1:49583)
0929 10:28:53.865921 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:53.865943 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:53.866043 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:53.866343 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:53.866345 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:53.866525 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:53.866621 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:53.866627 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:53.867543 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:53.867546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:53.868127 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:53.868135 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:53.868626 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:53.868630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:53.868703 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:53.869248 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:53.869259 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:53.871298 (+ 2039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":93,"threads_started":1}
W20260929 10:28:53.871698 30027 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:53.865681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:49583 (local address 127.20.92.62:40183)
0929 10:28:53.865997 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:53.866000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:53.866015 (+ 15us) server_negotiation.cc:407] Connection header received
0929 10:28:53.866132 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:53.866134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:53.866181 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:53.866262 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:53.866806 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:53.867404 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:53.868249 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:53.868484 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:53.868568 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:53.871501 (+ 2933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.1:49583: BlockingRecv error: recv got EOF from 127.20.92.1:49583 (error 108)
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":131,"threads_started":1}
Sep 29 10:28:54 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:54 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:54.089468 30028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:54.083683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.2:48845)
0929 10:28:54.083998 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:54.084016 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:54.084125 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:54.084400 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:54.084402 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:54.084606 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:54.084697 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:54.084703 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:54.085468 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:54.085475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:54.086063 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:54.086072 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:54.086536 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:54.086540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:54.086628 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:54.087167 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:54.087179 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:54.089337 (+ 2158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":108,"threads_started":1}
W20260929 10:28:54.089833 30027 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:54.083941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:48845 (local address 127.20.92.62:40183)
0929 10:28:54.084100 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:54.084105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:54.084118 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:28:54.084241 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:54.084244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:54.084294 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:54.084378 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:54.084822 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:54.085354 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:54.086189 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:54.086423 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:54.086531 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:54.089564 (+ 3033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.2:48845: BlockingRecv error: recv got EOF from 127.20.92.2:48845 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 10:28:54 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:54 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:54.288434 30029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:54.282533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.3:54711)
0929 10:28:54.282888 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:54.282903 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:54.283009 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:54.283286 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:54.283289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:54.283472 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:54.283567 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:54.283573 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:54.284552 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:54.284555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:54.285318 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:54.285328 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:54.285834 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:54.285837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:54.285911 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:54.286471 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:54.286483 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:54.288314 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":96,"threads_started":1}
W20260929 10:28:54.288856 30027 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:54.282643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:54711 (local address 127.20.92.62:40183)
0929 10:28:54.282886 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:54.282890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:54.282975 (+ 85us) server_negotiation.cc:407] Connection header received
0929 10:28:54.283092 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:54.283094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:54.283142 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:54.283231 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:54.283707 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:54.284428 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:54.285429 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:54.285669 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:54.285803 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:54.288742 (+ 2939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.3:54711: BlockingRecv error: recv got EOF from 127.20.92.3:54711 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 10:28:54 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:54 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:54.878249 30030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:54.872495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.1:33085)
0929 10:28:54.872822 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:54.872836 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:54.872927 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:54.873178 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:54.873188 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:54.873384 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:54.873476 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:54.873481 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:54.874396 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:54.874399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:54.874986 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:54.874995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:54.875502 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:54.875505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:54.875580 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:54.876096 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:54.876108 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:54.878118 (+ 2010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":73,"threads_started":1}
W20260929 10:28:54.878464 30031 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:54.872635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:33085 (local address 127.20.92.62:40183)
0929 10:28:54.872911 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:54.872915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:54.872930 (+ 15us) server_negotiation.cc:407] Connection header received
0929 10:28:54.873015 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:54.873018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:54.873060 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:54.873144 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:54.873613 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:54.874273 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:54.875128 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:54.875385 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:54.875479 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:54.878311 (+ 2832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.1:33085: BlockingRecv error: recv got EOF from 127.20.92.1:33085 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":80,"threads_started":1}
Sep 29 10:28:55 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:55 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:55.101933 30032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:55.090320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.2:52097)
0929 10:28:55.090617 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:55.090637 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:55.090772 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:55.091032 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:55.091035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:55.091281 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:55.091372 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:55.091378 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:55.092270 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:55.092277 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:55.092891 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:55.092900 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:55.093365 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:55.093369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:55.093447 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:55.099651 (+ 6204us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:55.099670 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:55.101725 (+ 2055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":95,"threads_started":1}
W20260929 10:28:55.102388 30031 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:55.090388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:52097 (local address 127.20.92.62:40183)
0929 10:28:55.090577 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:55.090581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:55.090731 (+ 150us) server_negotiation.cc:407] Connection header received
0929 10:28:55.090875 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:55.090878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:55.090927 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:55.091013 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:55.091508 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:55.092037 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:55.093016 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:55.093250 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:55.093362 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:55.102290 (+ 8928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.2:52097: BlockingRecv error: recv got EOF from 127.20.92.2:52097 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 29 10:28:55 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:55 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:55.295439 30033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:55.289404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.3:46779)
0929 10:28:55.289693 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:55.289706 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:55.289950 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:55.290269 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:55.290271 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:55.290446 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:55.290540 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:55.290545 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:55.291289 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:55.291292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:55.291882 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:55.291891 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:55.292403 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:55.292407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:55.292493 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:55.293090 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:55.293102 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:55.295310 (+ 2208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":110,"threads_started":1}
W20260929 10:28:55.295630 30031 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:55.289555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:46779 (local address 127.20.92.62:40183)
0929 10:28:55.289733 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:55.289737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:55.289861 (+ 124us) server_negotiation.cc:407] Connection header received
0929 10:28:55.290098 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:55.290101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:55.290152 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:55.290234 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:55.290641 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:55.291164 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:55.292030 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:55.292275 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:55.292396 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:55.295471 (+ 3075us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.3:46779: BlockingRecv error: recv got EOF from 127.20.92.3:46779 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 29 10:28:55 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:55 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:55.884779 30034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:55.879157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.1:44543)
0929 10:28:55.879437 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:55.879451 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:55.879540 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:55.879829 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:55.879831 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:55.880008 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:55.880102 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:55.880108 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:55.881060 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:55.881065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:55.881712 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:55.881723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:55.882219 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:55.882223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:55.882309 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:55.882836 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:55.882848 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:55.884663 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":99,"threads_started":1}
W20260929 10:28:55.884928 30035 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:55.879360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:44543 (local address 127.20.92.62:40183)
0929 10:28:55.879593 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:55.879597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:55.879611 (+ 14us) server_negotiation.cc:407] Connection header received
0929 10:28:55.879654 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:55.879656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:55.879697 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:55.879793 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:55.880279 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:55.880936 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:55.881864 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:55.882094 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:55.882202 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:55.884805 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.1:44543: BlockingRecv error: recv got EOF from 127.20.92.1:44543 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":83,"threads_started":1}
Sep 29 10:28:56 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:56 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:56.108279 30036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:56.102728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.2:45197)
0929 10:28:56.103003 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:56.103017 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:56.103110 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:56.103396 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:56.103397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:56.103573 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:56.103677 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:56.103683 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:56.104435 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:56.104437 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:56.104979 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:56.104985 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:56.105480 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:56.105483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:56.105546 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:56.106080 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:56.106092 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:56.108100 (+ 2008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":78,"threads_started":1}
W20260929 10:28:56.108325 30035 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:56.102890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:45197 (local address 127.20.92.62:40183)
0929 10:28:56.103037 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:56.103041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:56.103060 (+ 19us) server_negotiation.cc:407] Connection header received
0929 10:28:56.103229 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:56.103233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:56.103283 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:56.103368 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:56.103798 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:56.104333 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:56.105116 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:56.105368 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:56.105461 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:56.108233 (+ 2772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.2:45197: BlockingRecv error: recv got EOF from 127.20.92.2:45197 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 10:28:56 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:56 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:56.302196 30037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:56.296295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.3:37691)
0929 10:28:56.296608 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:56.296625 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:56.296780 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:56.297072 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:56.297075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:56.297276 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:56.297383 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:56.297388 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:56.298206 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:56.298210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:56.298836 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:56.298848 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:56.299363 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:56.299367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:56.299470 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:56.300047 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:56.300059 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:56.302068 (+ 2009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":86,"threads_started":1}
W20260929 10:28:56.302402 30035 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:56.296427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:37691 (local address 127.20.92.62:40183)
0929 10:28:56.296601 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:56.296606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:56.296713 (+ 107us) server_negotiation.cc:407] Connection header received
0929 10:28:56.296913 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:56.296915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:56.296961 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:56.297049 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:56.297505 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:56.298100 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:56.298961 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:56.299228 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:56.299343 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:56.302261 (+ 2918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.3:37691: BlockingRecv error: recv got EOF from 127.20.92.3:37691 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 29 10:28:56 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:56 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:56.891268 30038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:56.885574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.1:47215)
0929 10:28:56.885897 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:56.885911 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:56.886008 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:56.886245 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:56.886247 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:56.886443 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:56.886537 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:56.886542 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:56.887381 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:56.887384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:56.887939 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:56.887947 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:56.888393 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:56.888395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:56.888456 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:56.888961 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:56.888973 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:56.891139 (+ 2166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":74,"threads_started":1}
W20260929 10:28:56.891425 30039 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:56.885647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:47215 (local address 127.20.92.62:40183)
0929 10:28:56.885930 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:56.885934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:56.885960 (+ 26us) server_negotiation.cc:407] Connection header received
0929 10:28:56.886090 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:56.886092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:56.886134 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:56.886210 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:56.886713 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:56.887268 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:56.888053 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:56.888289 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:56.888375 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:56.891282 (+ 2907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.1:47215: BlockingRecv error: recv got EOF from 127.20.92.1:47215 (error 108)
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":134,"threads_started":1}
Sep 29 10:28:57 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:57 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:57.115026 30040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:57.109306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.2:47521)
0929 10:28:57.109649 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:57.109662 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:57.109820 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:57.110080 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:57.110083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:57.110302 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:57.110389 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:57.110395 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:57.111125 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:57.111130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:57.111681 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:57.111688 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:57.112132 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:57.112135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:57.112229 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:57.112725 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:57.112736 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:57.114863 (+ 2127us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":81,"threads_started":1}
W20260929 10:28:57.115172 30039 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:57.109440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:47521 (local address 127.20.92.62:40183)
0929 10:28:57.109581 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:57.109585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:57.109739 (+ 154us) server_negotiation.cc:407] Connection header received
0929 10:28:57.109918 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:57.109922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:57.109972 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:57.110059 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:57.110504 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:57.111016 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:57.111802 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:57.112023 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:57.112128 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:57.115026 (+ 2898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.2:47521: BlockingRecv error: recv got EOF from 127.20.92.2:47521 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 10:28:57 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:57 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:57.308163 30041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:57.303037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.3:47311)
0929 10:28:57.303295 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:57.303315 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:57.303410 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:57.303631 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:57.303633 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:57.303808 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:57.303902 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:57.303907 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:57.304624 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:57.304626 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:57.305192 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:57.305201 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:57.305637 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:57.305639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:57.305698 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:57.306231 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:57.306242 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:57.308048 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":95,"threads_started":1}
W20260929 10:28:57.308338 30039 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:57.303108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:47311 (local address 127.20.92.62:40183)
0929 10:28:57.303270 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:57.303274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:57.303381 (+ 107us) server_negotiation.cc:407] Connection header received
0929 10:28:57.303483 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:57.303486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:57.303529 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:57.303600 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:57.304001 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:57.304523 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:57.305321 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:57.305539 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:57.305627 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:57.308196 (+ 2569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.3:47311: BlockingRecv error: recv got EOF from 127.20.92.3:47311 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 10:28:57 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:57 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:57.902112 30042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:57.892299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.1:54113)
0929 10:28:57.892701 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:57.892718 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:57.892854 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:57.893159 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:57.893162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:57.893352 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:57.893456 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:57.893462 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:57.894312 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:57.894315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:57.894965 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:57.894980 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:57.895517 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:57.895521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:57.895598 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:57.899979 (+ 4381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:57.899993 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:57.901974 (+ 1981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":95,"threads_started":1}
W20260929 10:28:57.902333 30043 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:57.892401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:54113 (local address 127.20.92.62:40183)
0929 10:28:57.892744 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:57.892749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:57.892769 (+ 20us) server_negotiation.cc:407] Connection header received
0929 10:28:57.892946 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:57.892948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:57.892991 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:57.893070 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:57.893620 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:57.894192 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:57.895112 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:57.895381 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:57.895475 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:57.902141 (+ 6666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.1:54113: BlockingRecv error: recv got EOF from 127.20.92.1:54113 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":61,"threads_started":1}
Sep 29 10:28:58 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:58 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:58.121903 30044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:58.116035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.2:41653)
0929 10:28:58.116347 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:58.116364 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:58.116482 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:58.116742 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:58.116745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:58.116982 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:58.117071 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:58.117077 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:58.117859 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:58.117864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:58.118675 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:58.118686 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:58.119204 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:58.119208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:58.119293 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:58.119821 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:58.119834 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:58.121691 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":118,"threads_started":1}
W20260929 10:28:58.122040 30043 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:58.116192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:41653 (local address 127.20.92.62:40183)
0929 10:28:58.116361 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:58.116367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:58.116436 (+ 69us) server_negotiation.cc:407] Connection header received
0929 10:28:58.116594 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:58.116597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:58.116648 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:58.116724 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:58.117186 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:58.117719 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:58.118812 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:58.119067 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:58.119202 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:58.121924 (+ 2722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.2:41653: BlockingRecv error: recv got EOF from 127.20.92.2:41653 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 10:28:58 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:58 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:58.316809 30045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:58.309249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.3:56389)
0929 10:28:58.309858 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:58.309884 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:58.310221 (+ 337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:58.310544 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:58.310548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:58.310781 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:58.310888 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:58.310895 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:58.311866 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:58.311870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:58.312455 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:58.312464 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:58.312998 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:58.313001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:58.313104 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:58.313856 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:58.313871 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:58.316657 (+ 2786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":251,"threads_started":1}
W20260929 10:28:58.317229 30043 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:58.309552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:56389 (local address 127.20.92.62:40183)
0929 10:28:58.309858 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:58.309865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:58.310128 (+ 263us) server_negotiation.cc:407] Connection header received
0929 10:28:58.310349 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:58.310356 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:58.310424 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:58.310522 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:58.311022 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:58.311614 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:58.312578 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:58.312826 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:58.312946 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:58.317037 (+ 4091us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.3:56389: BlockingRecv error: recv got EOF from 127.20.92.3:56389 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
Sep 29 10:28:58 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:58 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:58.908692 30046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:58.902952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.1:60869)
0929 10:28:58.903245 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:58.903260 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:58.903352 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:58.903611 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:58.903613 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:58.903838 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:58.903949 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:58.903955 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:58.905006 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:58.905011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:58.905801 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:58.905811 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:58.906260 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:58.906262 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:58.906321 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:58.906782 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:58.906793 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:58.908572 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":86,"threads_started":1}
W20260929 10:28:58.908888 30047 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:58.903077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:60869 (local address 127.20.92.62:40183)
0929 10:28:58.903406 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:58.903410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:58.903423 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:28:58.903463 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:58.903465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:58.903505 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:58.903582 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:58.904088 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:58.904855 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:58.905928 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:58.906153 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:58.906240 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:58.908714 (+ 2474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.1:60869: BlockingRecv error: recv got EOF from 127.20.92.1:60869 (error 108)
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":106,"threads_started":1}
Sep 29 10:28:59 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:59 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:59.128994 30048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:59.122826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.2:34161)
0929 10:28:59.123226 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:59.123258 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:59.123397 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:59.123679 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:59.123682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:59.123922 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:59.124015 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:59.124021 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:59.124775 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:59.124780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:59.125370 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:59.125378 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:59.125874 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:59.125877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:59.125966 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:59.126541 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:59.126561 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:59.128796 (+ 2235us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":132,"threads_started":1}
W20260929 10:28:59.129078 30047 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:59.122988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:34161 (local address 127.20.92.62:40183)
0929 10:28:59.123175 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:59.123180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:59.123337 (+ 157us) server_negotiation.cc:407] Connection header received
0929 10:28:59.123517 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:59.123524 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:59.123573 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:59.123654 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:59.124131 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:59.124664 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:59.125487 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:59.125713 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:59.125870 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:59.128948 (+ 3078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.2:34161: BlockingRecv error: recv got EOF from 127.20.92.2:34161 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 29 10:28:59 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:59 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.3@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:59.323073 30049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:59.317604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.3:45215)
0929 10:28:59.318039 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:59.318052 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:59.318195 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:59.318462 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:59.318465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:59.318671 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:59.318758 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:59.318764 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:59.319495 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:59.319502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:59.320074 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:59.320081 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:59.320526 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:59.320530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:59.320616 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:59.321115 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:59.321142 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:59.322892 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":121,"threads_started":1}
W20260929 10:28:59.323127 30047 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:59.317701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:45215 (local address 127.20.92.62:40183)
0929 10:28:59.317967 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:59.317972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:59.318151 (+ 179us) server_negotiation.cc:407] Connection header received
0929 10:28:59.318303 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:59.318307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:59.318359 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:59.318443 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:59.318873 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:59.319393 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:59.320197 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:59.320420 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:59.320523 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:59.323013 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.3:45215: BlockingRecv error: recv got EOF from 127.20.92.3:45215 (error 108)
Metrics: {"server-negotiator.queue_time_us":174}
Sep 29 10:28:59 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:28:59 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.1@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:28:59.915441 30050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:28:59.909661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.1:48283)
0929 10:28:59.910063 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0929 10:28:59.910082 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 10:28:59.910201 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:28:59.910544 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:28:59.910547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:28:59.910728 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:28:59.910832 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:59.910837 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:59.911613 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:59.911616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:59.912210 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:28:59.912219 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:28:59.912661 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:28:59.912664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:28:59.912730 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:59.913298 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:28:59.913311 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:28:59.915319 (+ 2008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":188,"threads_started":1}
W20260929 10:28:59.915737 30051 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:28:59.909912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:48283 (local address 127.20.92.62:40183)
0929 10:28:59.910176 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 10:28:59.910180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:28:59.910195 (+ 15us) server_negotiation.cc:407] Connection header received
0929 10:28:59.910329 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:28:59.910331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:28:59.910379 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:28:59.910474 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:28:59.910944 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:59.911492 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:59.912321 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:28:59.912540 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:28:59.912629 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:28:59.915466 (+ 2837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.1:48283: BlockingRecv error: recv got EOF from 127.20.92.1:48283 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":100,"threads_started":1}
Sep 29 10:29:00 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:29:00 dist-test-slave-9cj5 krb5kdc[28993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.92.2@KRBTEST.COM for oryx/127.20.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:29:00.140923 30052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:00.130037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:40183 (local address 127.20.92.2:58043)
0929 10:29:00.130371 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0929 10:29:00.130384 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:29:00.130481 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:29:00.130792 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:29:00.130795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:29:00.130974 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:29:00.131064 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:00.131069 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:00.131963 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:00.131966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:00.132549 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:00.132558 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:00.133055 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:00.133058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:00.133120 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:00.138789 (+ 5669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:29:00.138807 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:29:00.140803 (+ 1996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.92.62:40183: Server oryx/127.20.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":112,"threads_started":1}
W20260929 10:29:00.141148 30051 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:29:00.130105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:58043 (local address 127.20.92.62:40183)
0929 10:29:00.130254 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:00.130259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:00.130484 (+ 225us) server_negotiation.cc:407] Connection header received
0929 10:29:00.130560 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:00.130562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:00.130662 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:00.130751 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:29:00.131204 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:00.131845 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:00.132663 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:00.132953 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:00.133040 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:00.141014 (+ 7974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.92.2:58043: BlockingRecv error: recv got EOF from 127.20.92.2:58043 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 10:29:00.199445 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 29576
I20260929 10:29:00.205226 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 29728
I20260929 10:29:00.211493 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 29860
I20260929 10:29:00.218999 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 29506
2026-09-29T10:29:00Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18180 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T10:29:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:29:00Z Disabled control of system clock
I20260929 10:29:00.252835 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:34119
--webserver_interface=127.20.92.62
--webserver_port=0
--builtin_ntp_servers=127.20.92.20:36873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:34119
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 10:29:00.368256 30056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:29:00.368562 30056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:29:00.368599 30056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:29:00.375205 30056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:29:00.375298 30056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:29:00.375315 30056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:29:00.375334 30056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:29:00.375349 30056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:29:00.380553 30056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:36873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:34119
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:34119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:29:00.382233 30056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:29:00.384024 30056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:29:00.394260 30061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:29:00.394309 30056 server_base.cc:1034] running on GCE node
W20260929 10:29:00.394328 30062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:29:00.394260 30064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:29:00.395267 30056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:29:00.396543 30056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:29:00.397809 30056 hybrid_clock.cc:648] HybridClock initialized: now 1790677740397723 us; error 50 us; skew 500 ppm
I20260929 10:29:00.399926 30056 webserver.cc:460] Webserver started at http://127.20.92.62:43281/ using document root <none> and password file <none>
I20260929 10:29:00.400542 30056 fs_manager.cc:359] Metadata directory not provided
I20260929 10:29:00.400668 30056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:29:00.400905 30056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:29:00.403137 30056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "3e0c95150ead4f11b9955fa02e95c276"
format_stamp: "Formatted at 2026-09-29 10:29:00 on dist-test-slave-9cj5"
server_key: "a4b2f6931a6ef4fffc48bee78137f72a"
server_key_iv: "70acd736327b9742fa21a8aa95723c36"
server_key_version: "encryptionkey@0"
I20260929 10:29:00.403872 30056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "3e0c95150ead4f11b9955fa02e95c276"
format_stamp: "Formatted at 2026-09-29 10:29:00 on dist-test-slave-9cj5"
server_key: "a4b2f6931a6ef4fffc48bee78137f72a"
server_key_iv: "70acd736327b9742fa21a8aa95723c36"
server_key_version: "encryptionkey@0"
I20260929 10:29:00.408354 30056 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:29:00.411876 30070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:00.414110 30056 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 10:29:00.414284 30056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "3e0c95150ead4f11b9955fa02e95c276"
format_stamp: "Formatted at 2026-09-29 10:29:00 on dist-test-slave-9cj5"
server_key: "a4b2f6931a6ef4fffc48bee78137f72a"
server_key_iv: "70acd736327b9742fa21a8aa95723c36"
server_key_version: "encryptionkey@0"
I20260929 10:29:00.414464 30056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:29:00.436975 30056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:29:00.437690 30056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:29:00.437943 30056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:29:00.445996 30056 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:34119
I20260929 10:29:00.446027 30122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:34119 every 8 connection(s)
I20260929 10:29:00.447126 30056 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:29:00.450932 30123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:29:00.451807 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 30056
I20260929 10:29:00.451907 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:29:00.452262 20848 external_mini_cluster.cc:1428] Setting key 8e98dcb93044ded5d66294cdab1ddd00
I20260929 10:29:00.458438 30123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e0c95150ead4f11b9955fa02e95c276: Bootstrap starting.
I20260929 10:29:00.460992 30123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e0c95150ead4f11b9955fa02e95c276: Neither blocks nor log segments found. Creating new log.
I20260929 10:29:00.461902 30123 log.cc:824] T 00000000000000000000000000000000 P 3e0c95150ead4f11b9955fa02e95c276: Log is configured to *not* fsync() on all Append() calls
I20260929 10:29:00.463205 30126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:00.453890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56700 (local address 127.20.92.62:34119)
0929 10:29:00.455083 (+ 1193us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:00.455108 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:00.455162 (+ 54us) server_negotiation.cc:407] Connection header received
0929 10:29:00.455989 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:00.456009 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:00.456457 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:00.456861 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:29:00.457326 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:00.458498 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:00.459583 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:00.460316 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:00.460508 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:00.461390 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:29:00.461395 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:29:00.461409 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:29:00.461871 (+ 462us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:29:00.461929 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:29:00.461951 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:29:00.462053 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 10:29:00.462211 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 10:29:00.462480 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":273,"threads_started":1}
I20260929 10:29:00.464308 30123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e0c95150ead4f11b9955fa02e95c276: No bootstrap required, opened a new log
I20260929 10:29:00.468753 30123 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e0c95150ead4f11b9955fa02e95c276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e0c95150ead4f11b9955fa02e95c276" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34119 } }
I20260929 10:29:00.469033 30123 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e0c95150ead4f11b9955fa02e95c276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:29:00.469074 30123 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e0c95150ead4f11b9955fa02e95c276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e0c95150ead4f11b9955fa02e95c276, State: Initialized, Role: FOLLOWER
I20260929 10:29:00.469568 30123 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e0c95150ead4f11b9955fa02e95c276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e0c95150ead4f11b9955fa02e95c276" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34119 } }
I20260929 10:29:00.469707 30123 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e0c95150ead4f11b9955fa02e95c276 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:29:00.469833 30123 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e0c95150ead4f11b9955fa02e95c276 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:29:00.469959 30123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e0c95150ead4f11b9955fa02e95c276 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:29:00.471045 30123 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e0c95150ead4f11b9955fa02e95c276 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e0c95150ead4f11b9955fa02e95c276" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34119 } }
I20260929 10:29:00.471441 30123 leader_election.cc:302] T 00000000000000000000000000000000 P 3e0c95150ead4f11b9955fa02e95c276 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e0c95150ead4f11b9955fa02e95c276; no voters:
I20260929 10:29:00.471802 30123 leader_election.cc:288] T 00000000000000000000000000000000 P 3e0c95150ead4f11b9955fa02e95c276 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:29:00.472227 30128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e0c95150ead4f11b9955fa02e95c276 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:29:00.472584 30128 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e0c95150ead4f11b9955fa02e95c276 [term 1 LEADER]: Becoming Leader. State: Replica: 3e0c95150ead4f11b9955fa02e95c276, State: Running, Role: LEADER
I20260929 10:29:00.472971 30128 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e0c95150ead4f11b9955fa02e95c276 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e0c95150ead4f11b9955fa02e95c276" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34119 } }
I20260929 10:29:00.473884 30123 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e0c95150ead4f11b9955fa02e95c276 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:29:00.474781 30129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e0c95150ead4f11b9955fa02e95c276 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e0c95150ead4f11b9955fa02e95c276" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e0c95150ead4f11b9955fa02e95c276" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34119 } } }
I20260929 10:29:00.475054 30130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e0c95150ead4f11b9955fa02e95c276 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e0c95150ead4f11b9955fa02e95c276. Latest consensus state: current_term: 1 leader_uuid: "3e0c95150ead4f11b9955fa02e95c276" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e0c95150ead4f11b9955fa02e95c276" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34119 } } }
I20260929 10:29:00.475317 30130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e0c95150ead4f11b9955fa02e95c276 [sys.catalog]: This master's current role is: LEADER
I20260929 10:29:00.475319 30129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e0c95150ead4f11b9955fa02e95c276 [sys.catalog]: This master's current role is: LEADER
I20260929 10:29:00.475836 30137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:29:00.479612 30137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:29:00.487255 30137 catalog_manager.cc:1347] Generated new cluster ID: c576773be4db4f10a8c60f8f8445518f
I20260929 10:29:00.487393 30137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:29:00.503485 30137 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:29:00.504555 30137 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:29:00.513615 30137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e0c95150ead4f11b9955fa02e95c276: Generated new TSK 0
I20260929 10:29:00.514566 30137 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:29:00.523801 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:0
--local_ip_for_outbound_sockets=127.20.92.1
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:34119
--builtin_ntp_servers=127.20.92.20:36873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 10:29:00.653986 30147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:29:00.654356 30147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:29:00.654439 30147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:29:00.659351 30147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:29:00.659483 30147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:29:00.659592 30147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:29:00.665560 30147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:36873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:34119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:29:00.666807 30147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:29:00.668272 30147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:29:00.679147 30147 server_base.cc:1034] running on GCE node
W20260929 10:29:00.679075 30153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:29:00.679059 30155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:29:00.679064 30152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:29:00.679744 30147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:29:00.680395 30147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:29:00.681628 30147 hybrid_clock.cc:648] HybridClock initialized: now 1790677740681571 us; error 40 us; skew 500 ppm
I20260929 10:29:00.683980 30147 webserver.cc:460] Webserver started at http://127.20.92.1:42831/ using document root <none> and password file <none>
I20260929 10:29:00.684628 30147 fs_manager.cc:359] Metadata directory not provided
I20260929 10:29:00.684702 30147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:29:00.684895 30147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:29:00.687386 30147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/data/instance:
uuid: "9d8dfae0dae64bd6b9b972a55480ba3b"
format_stamp: "Formatted at 2026-09-29 10:29:00 on dist-test-slave-9cj5"
server_key: "58cdd11e39fbc316e624e437f8368ecd"
server_key_iv: "64f5b989cfb6eed6851b57cabf097755"
server_key_version: "encryptionkey@0"
I20260929 10:29:00.688159 30147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance:
uuid: "9d8dfae0dae64bd6b9b972a55480ba3b"
format_stamp: "Formatted at 2026-09-29 10:29:00 on dist-test-slave-9cj5"
server_key: "58cdd11e39fbc316e624e437f8368ecd"
server_key_iv: "64f5b989cfb6eed6851b57cabf097755"
server_key_version: "encryptionkey@0"
I20260929 10:29:00.692566 30147 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 10:29:00.695519 30161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:00.697329 30147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 10:29:00.697459 30147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "9d8dfae0dae64bd6b9b972a55480ba3b"
format_stamp: "Formatted at 2026-09-29 10:29:00 on dist-test-slave-9cj5"
server_key: "58cdd11e39fbc316e624e437f8368ecd"
server_key_iv: "64f5b989cfb6eed6851b57cabf097755"
server_key_version: "encryptionkey@0"
I20260929 10:29:00.697582 30147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:29:00.719545 30147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:29:00.720371 30147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:29:00.720553 30147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:29:00.721185 30147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:29:00.722287 30147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:29:00.722347 30147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:00.722390 30147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:29:00.722409 30147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:00.732479 30147 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:46815
I20260929 10:29:00.732502 30274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:46815 every 8 connection(s)
I20260929 10:29:00.733512 30147 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
I20260929 10:29:00.742120 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 30147
I20260929 10:29:00.742228 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance
I20260929 10:29:00.742559 20848 external_mini_cluster.cc:1428] Setting key 72e7fb3413d1e93ccc0ece1dd21ca4e7
I20260929 10:29:00.744601 30126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:00.735483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:33475 (local address 127.20.92.62:34119)
0929 10:29:00.735644 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:00.735649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:00.737038 (+ 1389us) server_negotiation.cc:407] Connection header received
0929 10:29:00.738681 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:00.738684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:00.738740 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:00.738839 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:29:00.739926 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:00.740556 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:00.742040 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:00.742389 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:00.742517 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:00.743945 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:29:00.743948 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:29:00.743951 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:29:00.743963 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:29:00.743983 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:29:00.743991 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:29:00.744116 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 10:29:00.744365 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 10:29:00.744443 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 10:29:00.745045 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:0
--local_ip_for_outbound_sockets=127.20.92.2
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:34119
--builtin_ntp_servers=127.20.92.20:36873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 10:29:00.745872 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:00.735987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:34119 (local address 127.20.92.1:33475)
0929 10:29:00.736812 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0929 10:29:00.736871 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 10:29:00.738412 (+ 1541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:29:00.739025 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:29:00.739033 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:29:00.739137 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:29:00.739718 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:00.739734 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:00.740740 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:00.740744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:00.741897 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:00.741909 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:00.742569 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:00.742574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:00.742788 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:00.743584 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:29:00.743586 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:29:00.743649 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:29:00.743748 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:29:00.744151 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:29:00.744162 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:29:00.744169 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 10:29:00.744334 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 10:29:00.744626 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":164,"threads_started":1}
I20260929 10:29:00.747457 30275 heartbeater.cc:347] Connected to a master server at 127.20.92.62:34119
I20260929 10:29:00.747743 30275 heartbeater.cc:464] Registering TS with master...
I20260929 10:29:00.748318 30275 heartbeater.cc:511] Master 127.20.92.62:34119 requested a full tablet report, sending...
I20260929 10:29:00.750065 30087 ts_manager.cc:194] Registered new tserver with Master: 9d8dfae0dae64bd6b9b972a55480ba3b (127.20.92.1:46815)
I20260929 10:29:00.751238 30087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.92.1:33475
W20260929 10:29:00.866156 30278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:29:00.866460 30278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:29:00.866500 30278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:29:00.870814 30278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:29:00.870903 30278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:29:00.870962 30278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:29:00.876614 30278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:36873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:34119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:29:00.877866 30278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:29:00.879307 30278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:29:00.889962 30284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:29:00.889950 30283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:29:00.890010 30278 server_base.cc:1034] running on GCE node
W20260929 10:29:00.889950 30286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:29:00.890724 30278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:29:00.891364 30278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:29:00.892546 30278 hybrid_clock.cc:648] HybridClock initialized: now 1790677740892519 us; error 90 us; skew 500 ppm
I20260929 10:29:00.894564 30278 webserver.cc:460] Webserver started at http://127.20.92.2:35221/ using document root <none> and password file <none>
I20260929 10:29:00.895191 30278 fs_manager.cc:359] Metadata directory not provided
I20260929 10:29:00.895259 30278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:29:00.895481 30278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:29:00.897732 30278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/data/instance:
uuid: "fbcd90a089a84d5791580a851afee0ee"
format_stamp: "Formatted at 2026-09-29 10:29:00 on dist-test-slave-9cj5"
server_key: "25662cd541fd35dec1e9f62f7cbebd18"
server_key_iv: "9960a86b822ddde48174272eb7b8d956"
server_key_version: "encryptionkey@0"
I20260929 10:29:00.898427 30278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance:
uuid: "fbcd90a089a84d5791580a851afee0ee"
format_stamp: "Formatted at 2026-09-29 10:29:00 on dist-test-slave-9cj5"
server_key: "25662cd541fd35dec1e9f62f7cbebd18"
server_key_iv: "9960a86b822ddde48174272eb7b8d956"
server_key_version: "encryptionkey@0"
I20260929 10:29:00.902480 30278 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 10:29:00.905350 30292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:00.906883 30278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 10:29:00.906996 30278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "fbcd90a089a84d5791580a851afee0ee"
format_stamp: "Formatted at 2026-09-29 10:29:00 on dist-test-slave-9cj5"
server_key: "25662cd541fd35dec1e9f62f7cbebd18"
server_key_iv: "9960a86b822ddde48174272eb7b8d956"
server_key_version: "encryptionkey@0"
I20260929 10:29:00.907153 30278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:29:00.951303 30278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:29:00.952087 30278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:29:00.952306 30278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:29:00.952960 30278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:29:00.954048 30278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:29:00.954099 30278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:00.954142 30278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:29:00.954159 30278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:00.964401 30278 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:43149
I20260929 10:29:00.964505 30405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:43149 every 8 connection(s)
I20260929 10:29:00.965358 30278 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
I20260929 10:29:00.974512 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 30278
I20260929 10:29:00.974534 30126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:00.967228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:56787 (local address 127.20.92.62:34119)
0929 10:29:00.967378 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:00.967382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:00.968434 (+ 1052us) server_negotiation.cc:407] Connection header received
0929 10:29:00.969873 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:00.969876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:00.969925 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:00.970000 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:29:00.970886 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:00.971413 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:00.972542 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:00.972748 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:00.972881 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:00.974003 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:29:00.974006 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:29:00.974008 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:29:00.974020 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:29:00.974038 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:29:00.974045 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:29:00.974103 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 10:29:00.974318 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 10:29:00.974385 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 10:29:00.974638 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance
I20260929 10:29:00.974975 20848 external_mini_cluster.cc:1428] Setting key 0f4c06ff6bd71ff4ebc3dc0556949732
I20260929 10:29:00.975485 30408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:00.967662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:34119 (local address 127.20.92.2:56787)
0929 10:29:00.968280 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0929 10:29:00.968321 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 10:29:00.969612 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:29:00.970155 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:29:00.970162 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:29:00.970245 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:29:00.970735 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:00.970747 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:00.971521 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:00.971524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:00.972439 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:00.972447 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:00.972926 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:00.972929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:00.973069 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:00.973706 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:29:00.973708 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:29:00.973789 (+ 81us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:29:00.973866 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:29:00.974151 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:29:00.974158 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:29:00.974162 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:29:00.974292 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 10:29:00.974567 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":109,"threads_started":1}
I20260929 10:29:00.976894 30406 heartbeater.cc:347] Connected to a master server at 127.20.92.62:34119
I20260929 10:29:00.977150 30406 heartbeater.cc:464] Registering TS with master...
I20260929 10:29:00.977303 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:0
--local_ip_for_outbound_sockets=127.20.92.3
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:34119
--builtin_ntp_servers=127.20.92.20:36873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 10:29:00.977721 30406 heartbeater.cc:511] Master 127.20.92.62:34119 requested a full tablet report, sending...
I20260929 10:29:00.978808 30087 ts_manager.cc:194] Registered new tserver with Master: fbcd90a089a84d5791580a851afee0ee (127.20.92.2:43149)
I20260929 10:29:00.979622 30087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.92.2:56787
W20260929 10:29:01.091967 30409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:29:01.092280 30409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:29:01.092319 30409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:29:01.096729 30409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:29:01.096812 30409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:29:01.096869 30409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:29:01.101979 30409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:36873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:34119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:29:01.103245 30409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:29:01.104607 30409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:29:01.114140 30415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:29:01.114131 30414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:29:01.114217 30417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:29:01.114640 30409 server_base.cc:1034] running on GCE node
I20260929 10:29:01.115096 30409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:29:01.115898 30409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:29:01.117095 30409 hybrid_clock.cc:648] HybridClock initialized: now 1790677741117056 us; error 51 us; skew 500 ppm
I20260929 10:29:01.119171 30409 webserver.cc:460] Webserver started at http://127.20.92.3:45615/ using document root <none> and password file <none>
I20260929 10:29:01.119815 30409 fs_manager.cc:359] Metadata directory not provided
I20260929 10:29:01.119910 30409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:29:01.120143 30409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:29:01.122463 30409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/data/instance:
uuid: "798fd19fc9d3443e89dc890e85b74f7c"
format_stamp: "Formatted at 2026-09-29 10:29:01 on dist-test-slave-9cj5"
server_key: "b8d396c61652049f22177712b67081fc"
server_key_iv: "d724ae4e12f902b5dcf50467bb4079cc"
server_key_version: "encryptionkey@0"
I20260929 10:29:01.123044 30409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance:
uuid: "798fd19fc9d3443e89dc890e85b74f7c"
format_stamp: "Formatted at 2026-09-29 10:29:01 on dist-test-slave-9cj5"
server_key: "b8d396c61652049f22177712b67081fc"
server_key_iv: "d724ae4e12f902b5dcf50467bb4079cc"
server_key_version: "encryptionkey@0"
I20260929 10:29:01.127002 30409 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 10:29:01.130028 30423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:01.131372 30409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:29:01.131510 30409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "798fd19fc9d3443e89dc890e85b74f7c"
format_stamp: "Formatted at 2026-09-29 10:29:01 on dist-test-slave-9cj5"
server_key: "b8d396c61652049f22177712b67081fc"
server_key_iv: "d724ae4e12f902b5dcf50467bb4079cc"
server_key_version: "encryptionkey@0"
I20260929 10:29:01.131634 30409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:29:01.141326 30409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:29:01.142119 30409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:29:01.142311 30409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:29:01.142880 30409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:29:01.143842 30409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:29:01.143908 30409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:01.143965 30409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:29:01.144018 30409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:01.153344 30409 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:38475
I20260929 10:29:01.153415 30536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:38475 every 8 connection(s)
I20260929 10:29:01.154371 30409 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:29:01.164186 30126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:01.156426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:58079 (local address 127.20.92.62:34119)
0929 10:29:01.156572 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:01.156575 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:01.157599 (+ 1024us) server_negotiation.cc:407] Connection header received
0929 10:29:01.159118 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:01.159121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:01.159172 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:01.159340 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:29:01.160425 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:01.160948 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:01.162206 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:01.162453 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:01.162538 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:01.163771 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:29:01.163774 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:29:01.163776 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:29:01.163790 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:29:01.163814 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:29:01.163823 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:29:01.163883 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 10:29:01.164002 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 10:29:01.164060 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 10:29:01.164453 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 30409
I20260929 10:29:01.164573 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance
I20260929 10:29:01.164882 20848 external_mini_cluster.cc:1428] Setting key 92f9bcec3c782eb5083d5d389c5aabd6
I20260929 10:29:01.165307 30539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:01.156742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:34119 (local address 127.20.92.3:58079)
0929 10:29:01.157427 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0929 10:29:01.157467 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 10:29:01.158875 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:29:01.159546 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:29:01.159554 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:29:01.159706 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:29:01.160260 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:01.160277 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:01.161102 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:01.161105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:01.162077 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:01.162087 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:01.162605 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:01.162608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:01.162764 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:01.163509 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:29:01.163510 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:29:01.163551 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:29:01.163618 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:29:01.163882 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:29:01.163886 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:29:01.163890 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:29:01.163978 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 10:29:01.164289 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":117,"threads_started":1}
I20260929 10:29:01.166610 30537 heartbeater.cc:347] Connected to a master server at 127.20.92.62:34119
I20260929 10:29:01.166872 30537 heartbeater.cc:464] Registering TS with master...
I20260929 10:29:01.167439 30537 heartbeater.cc:511] Master 127.20.92.62:34119 requested a full tablet report, sending...
I20260929 10:29:01.168463 30087 ts_manager.cc:194] Registered new tserver with Master: 798fd19fc9d3443e89dc890e85b74f7c (127.20.92.3:38475)
I20260929 10:29:01.169166 30087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.92.3:58079
I20260929 10:29:01.179083 20848 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 10:29:01.182186 30126 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:29:01.181115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56712 (local address 127.20.92.62:34119)
0929 10:29:01.181303 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:01.181307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:01.181480 (+ 173us) server_negotiation.cc:407] Connection header received
0929 10:29:01.181526 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:01.181528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:01.181572 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:01.181659 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:29:01.182045 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56712: BlockingRecv error: recv got EOF from 127.0.0.1:56712 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 10:29:01.182399 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 30147
I20260929 10:29:01.188930 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 30278
I20260929 10:29:01.194890 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 30409
I20260929 10:29:01.201097 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 30056
2026-09-29T10:29:01Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (981 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T10:29:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:29:01Z Disabled control of system clock
I20260929 10:29:01.233433 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:45207
--webserver_interface=127.20.92.62
--webserver_port=0
--builtin_ntp_servers=127.20.92.20:37547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:45207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 10:29:01.352916 30548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:29:01.353209 30548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:29:01.353248 30548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:29:01.357425 30548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:29:01.357499 30548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:29:01.357515 30548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:29:01.357532 30548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:29:01.357547 30548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:29:01.362771 30548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:37547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:45207
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:45207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:29:01.363929 30548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:29:01.365322 30548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:29:01.374228 30556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:29:01.374396 30548 server_base.cc:1034] running on GCE node
W20260929 10:29:01.374289 30553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:29:01.374228 30554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:29:01.375379 30548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:29:01.376672 30548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:29:01.377887 30548 hybrid_clock.cc:648] HybridClock initialized: now 1790677741377842 us; error 57 us; skew 500 ppm
I20260929 10:29:01.380045 30548 webserver.cc:460] Webserver started at http://127.20.92.62:42591/ using document root <none> and password file <none>
I20260929 10:29:01.380683 30548 fs_manager.cc:359] Metadata directory not provided
I20260929 10:29:01.380761 30548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:29:01.380992 30548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:29:01.383385 30548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "995ebf355cad4ea4b5ff4f63965be32c"
format_stamp: "Formatted at 2026-09-29 10:29:01 on dist-test-slave-9cj5"
server_key: "8353f1eabae1dcfedc70efe0c538179d"
server_key_iv: "1811c0944e246e224572b55e8855d838"
server_key_version: "encryptionkey@0"
I20260929 10:29:01.384064 30548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "995ebf355cad4ea4b5ff4f63965be32c"
format_stamp: "Formatted at 2026-09-29 10:29:01 on dist-test-slave-9cj5"
server_key: "8353f1eabae1dcfedc70efe0c538179d"
server_key_iv: "1811c0944e246e224572b55e8855d838"
server_key_version: "encryptionkey@0"
I20260929 10:29:01.388626 30548 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:29:01.391618 30562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:01.393299 30548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:29:01.393473 30548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "995ebf355cad4ea4b5ff4f63965be32c"
format_stamp: "Formatted at 2026-09-29 10:29:01 on dist-test-slave-9cj5"
server_key: "8353f1eabae1dcfedc70efe0c538179d"
server_key_iv: "1811c0944e246e224572b55e8855d838"
server_key_version: "encryptionkey@0"
I20260929 10:29:01.393610 30548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:29:01.398736 30548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:29:01.399371 30548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:29:01.399564 30548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:29:01.406597 30548 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:45207
I20260929 10:29:01.406625 30614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:45207 every 8 connection(s)
I20260929 10:29:01.407572 30548 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:29:01.410351 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 30548
I20260929 10:29:01.410596 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:29:01.410998 30615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:29:01.411101 20848 external_mini_cluster.cc:1428] Setting key a979dbc090cbf6d4f65ac5caef123db7
I20260929 10:29:01.417200 30618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:01.412506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35620 (local address 127.20.92.62:45207)
0929 10:29:01.413227 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:01.413245 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:01.413289 (+ 44us) server_negotiation.cc:407] Connection header received
0929 10:29:01.414039 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:01.414063 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:01.414513 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:01.414888 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:29:01.415680 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:29:01.415686 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:29:01.415700 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:29:01.415711 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:29:01.415780 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:29:01.415814 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:29:01.415958 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 10:29:01.416033 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 10:29:01.416269 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":131,"threads_started":1}
I20260929 10:29:01.418680 30615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 995ebf355cad4ea4b5ff4f63965be32c: Bootstrap starting.
I20260929 10:29:01.420924 30615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 995ebf355cad4ea4b5ff4f63965be32c: Neither blocks nor log segments found. Creating new log.
I20260929 10:29:01.421873 30615 log.cc:824] T 00000000000000000000000000000000 P 995ebf355cad4ea4b5ff4f63965be32c: Log is configured to *not* fsync() on all Append() calls
I20260929 10:29:01.424078 30615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 995ebf355cad4ea4b5ff4f63965be32c: No bootstrap required, opened a new log
I20260929 10:29:01.428277 30615 raft_consensus.cc:348] T 00000000000000000000000000000000 P 995ebf355cad4ea4b5ff4f63965be32c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "995ebf355cad4ea4b5ff4f63965be32c" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45207 } }
I20260929 10:29:01.428540 30615 raft_consensus.cc:374] T 00000000000000000000000000000000 P 995ebf355cad4ea4b5ff4f63965be32c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:29:01.428577 30615 raft_consensus.cc:729] T 00000000000000000000000000000000 P 995ebf355cad4ea4b5ff4f63965be32c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 995ebf355cad4ea4b5ff4f63965be32c, State: Initialized, Role: FOLLOWER
I20260929 10:29:01.429138 30615 consensus_queue.cc:260] T 00000000000000000000000000000000 P 995ebf355cad4ea4b5ff4f63965be32c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "995ebf355cad4ea4b5ff4f63965be32c" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45207 } }
I20260929 10:29:01.429271 30615 raft_consensus.cc:388] T 00000000000000000000000000000000 P 995ebf355cad4ea4b5ff4f63965be32c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:29:01.429359 30615 raft_consensus.cc:482] T 00000000000000000000000000000000 P 995ebf355cad4ea4b5ff4f63965be32c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:29:01.429471 30615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 995ebf355cad4ea4b5ff4f63965be32c [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:29:01.430538 30615 raft_consensus.cc:504] T 00000000000000000000000000000000 P 995ebf355cad4ea4b5ff4f63965be32c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "995ebf355cad4ea4b5ff4f63965be32c" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45207 } }
I20260929 10:29:01.430922 30615 leader_election.cc:302] T 00000000000000000000000000000000 P 995ebf355cad4ea4b5ff4f63965be32c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 995ebf355cad4ea4b5ff4f63965be32c; no voters:
I20260929 10:29:01.431267 30615 leader_election.cc:288] T 00000000000000000000000000000000 P 995ebf355cad4ea4b5ff4f63965be32c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:29:01.431481 30620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 995ebf355cad4ea4b5ff4f63965be32c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:29:01.431789 30620 raft_consensus.cc:686] T 00000000000000000000000000000000 P 995ebf355cad4ea4b5ff4f63965be32c [term 1 LEADER]: Becoming Leader. State: Replica: 995ebf355cad4ea4b5ff4f63965be32c, State: Running, Role: LEADER
I20260929 10:29:01.432183 30620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 995ebf355cad4ea4b5ff4f63965be32c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "995ebf355cad4ea4b5ff4f63965be32c" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45207 } }
I20260929 10:29:01.432675 30615 sys_catalog.cc:564] T 00000000000000000000000000000000 P 995ebf355cad4ea4b5ff4f63965be32c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:29:01.433936 30622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 995ebf355cad4ea4b5ff4f63965be32c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 995ebf355cad4ea4b5ff4f63965be32c. Latest consensus state: current_term: 1 leader_uuid: "995ebf355cad4ea4b5ff4f63965be32c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "995ebf355cad4ea4b5ff4f63965be32c" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45207 } } }
I20260929 10:29:01.434072 30622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 995ebf355cad4ea4b5ff4f63965be32c [sys.catalog]: This master's current role is: LEADER
I20260929 10:29:01.434449 30621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 995ebf355cad4ea4b5ff4f63965be32c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "995ebf355cad4ea4b5ff4f63965be32c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "995ebf355cad4ea4b5ff4f63965be32c" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45207 } } }
I20260929 10:29:01.434630 30621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 995ebf355cad4ea4b5ff4f63965be32c [sys.catalog]: This master's current role is: LEADER
I20260929 10:29:01.434537 30629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:29:01.437937 30629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:29:01.443337 30629 catalog_manager.cc:1347] Generated new cluster ID: a13068e5280445fbb1c8c493399618c4
I20260929 10:29:01.443444 30629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:29:01.461971 30629 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:29:01.462369 30629 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:29:01.473132 30629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 995ebf355cad4ea4b5ff4f63965be32c: Generated new TSK 0
I20260929 10:29:01.473996 30629 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:29:01.479817 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:0
--local_ip_for_outbound_sockets=127.20.92.1
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:45207
--builtin_ntp_servers=127.20.92.20:37547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 10:29:01.621313 30639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:29:01.621623 30639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:29:01.621721 30639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:29:01.626437 30639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:29:01.626518 30639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:29:01.626585 30639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:29:01.632227 30639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:37547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:45207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:29:01.633725 30639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:29:01.635383 30639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:29:01.646109 30647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:29:01.646098 30644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:29:01.646098 30645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:29:01.646435 30639 server_base.cc:1034] running on GCE node
I20260929 10:29:01.646903 30639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:29:01.647586 30639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:29:01.648798 30639 hybrid_clock.cc:648] HybridClock initialized: now 1790677741648783 us; error 26 us; skew 500 ppm
I20260929 10:29:01.651057 30639 webserver.cc:460] Webserver started at http://127.20.92.1:45799/ using document root <none> and password file <none>
I20260929 10:29:01.651784 30639 fs_manager.cc:359] Metadata directory not provided
I20260929 10:29:01.651890 30639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:29:01.652134 30639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:29:01.654356 30639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/data/instance:
uuid: "189a4512741f4143b9ec9f4aea445a45"
format_stamp: "Formatted at 2026-09-29 10:29:01 on dist-test-slave-9cj5"
server_key: "988cf4ce51e26529c9e00beff864c0b7"
server_key_iv: "1be134f8f6113d403f78277ec0adc5e6"
server_key_version: "encryptionkey@0"
I20260929 10:29:01.654953 30639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance:
uuid: "189a4512741f4143b9ec9f4aea445a45"
format_stamp: "Formatted at 2026-09-29 10:29:01 on dist-test-slave-9cj5"
server_key: "988cf4ce51e26529c9e00beff864c0b7"
server_key_iv: "1be134f8f6113d403f78277ec0adc5e6"
server_key_version: "encryptionkey@0"
I20260929 10:29:01.658788 30639 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 10:29:01.661509 30653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:01.663076 30639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 10:29:01.663233 30639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "189a4512741f4143b9ec9f4aea445a45"
format_stamp: "Formatted at 2026-09-29 10:29:01 on dist-test-slave-9cj5"
server_key: "988cf4ce51e26529c9e00beff864c0b7"
server_key_iv: "1be134f8f6113d403f78277ec0adc5e6"
server_key_version: "encryptionkey@0"
I20260929 10:29:01.663369 30639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:29:01.668121 30639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:29:01.668738 30639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:29:01.668928 30639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:29:01.669461 30639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:29:01.670419 30639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:29:01.670487 30639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:01.670558 30639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:29:01.670601 30639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:01.680186 30639 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:37345
I20260929 10:29:01.680224 30766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:37345 every 8 connection(s)
I20260929 10:29:01.681137 30639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
I20260929 10:29:01.687952 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 30639
I20260929 10:29:01.688170 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance
I20260929 10:29:01.688563 20848 external_mini_cluster.cc:1428] Setting key b2a6dee47bc84f03e3ca21c5d24eea9d
I20260929 10:29:01.689064 30618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:01.683399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:42879 (local address 127.20.92.62:45207)
0929 10:29:01.683632 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:01.683636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:01.685005 (+ 1369us) server_negotiation.cc:407] Connection header received
0929 10:29:01.686977 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:01.686981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:01.687033 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:01.687153 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:29:01.688554 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:29:01.688557 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:29:01.688559 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:29:01.688572 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:29:01.688594 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:29:01.688602 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:29:01.688720 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 10:29:01.688864 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 10:29:01.688918 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 10:29:01.690316 30769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:01.683840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45207 (local address 127.20.92.1:42879)
0929 10:29:01.684794 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0929 10:29:01.684847 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 10:29:01.686685 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:29:01.687350 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:29:01.687358 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:29:01.687458 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:29:01.688211 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:29:01.688213 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:29:01.688292 (+ 79us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:29:01.688384 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:29:01.688700 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:29:01.688709 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:29:01.688715 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 10:29:01.688915 (+ 200us) client_negotiation.cc:240] Negotiation successful
0929 10:29:01.689214 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":134,"threads_started":1}
I20260929 10:29:01.691356 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:0
--local_ip_for_outbound_sockets=127.20.92.2
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:45207
--builtin_ntp_servers=127.20.92.20:37547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 10:29:01.691839 30767 heartbeater.cc:347] Connected to a master server at 127.20.92.62:45207
I20260929 10:29:01.692348 30767 heartbeater.cc:464] Registering TS with master...
I20260929 10:29:01.693017 30767 heartbeater.cc:511] Master 127.20.92.62:45207 requested a full tablet report, sending...
I20260929 10:29:01.695576 30579 ts_manager.cc:194] Registered new tserver with Master: 189a4512741f4143b9ec9f4aea445a45 (127.20.92.1:37345)
W20260929 10:29:01.815353 30770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:29:01.815644 30770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:29:01.815718 30770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:29:01.820686 30770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:29:01.820775 30770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:29:01.820844 30770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:29:01.826684 30770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:37547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:45207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:29:01.827935 30770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:29:01.829425 30770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:29:01.840794 30776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:29:01.840952 30775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:29:01.841085 30770 server_base.cc:1034] running on GCE node
W20260929 10:29:01.840817 30778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:29:01.841792 30770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:29:01.842526 30770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:29:01.843881 30770 hybrid_clock.cc:648] HybridClock initialized: now 1790677741843863 us; error 36 us; skew 500 ppm
I20260929 10:29:01.846088 30770 webserver.cc:460] Webserver started at http://127.20.92.2:39641/ using document root <none> and password file <none>
I20260929 10:29:01.846763 30770 fs_manager.cc:359] Metadata directory not provided
I20260929 10:29:01.846865 30770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:29:01.847101 30770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:29:01.849568 30770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/data/instance:
uuid: "bfb8727ec6884f2f971d8ac1fbcf3aa0"
format_stamp: "Formatted at 2026-09-29 10:29:01 on dist-test-slave-9cj5"
server_key: "a86943b843c958d6198e241737149b1f"
server_key_iv: "b1cb31631f39067c87b5e09dc8b5b027"
server_key_version: "encryptionkey@0"
I20260929 10:29:01.850273 30770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance:
uuid: "bfb8727ec6884f2f971d8ac1fbcf3aa0"
format_stamp: "Formatted at 2026-09-29 10:29:01 on dist-test-slave-9cj5"
server_key: "a86943b843c958d6198e241737149b1f"
server_key_iv: "b1cb31631f39067c87b5e09dc8b5b027"
server_key_version: "encryptionkey@0"
I20260929 10:29:01.854586 30770 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 10:29:01.858444 30784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:01.860544 30770 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 10:29:01.860746 30770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "bfb8727ec6884f2f971d8ac1fbcf3aa0"
format_stamp: "Formatted at 2026-09-29 10:29:01 on dist-test-slave-9cj5"
server_key: "a86943b843c958d6198e241737149b1f"
server_key_iv: "b1cb31631f39067c87b5e09dc8b5b027"
server_key_version: "encryptionkey@0"
I20260929 10:29:01.860916 30770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:29:01.866371 30770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:29:01.867142 30770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:29:01.867367 30770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:29:01.868053 30770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:29:01.869107 30770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:29:01.869175 30770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:01.869246 30770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:29:01.869297 30770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:01.882808 30770 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:41913
I20260929 10:29:01.882881 30897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:41913 every 8 connection(s)
I20260929 10:29:01.883874 30770 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
I20260929 10:29:01.889650 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 30770
I20260929 10:29:01.889861 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance
I20260929 10:29:01.890264 20848 external_mini_cluster.cc:1428] Setting key 8243699269e372fc33a40e3d1d3eb135
I20260929 10:29:01.893187 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:0
--local_ip_for_outbound_sockets=127.20.92.3
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:45207
--builtin_ntp_servers=127.20.92.20:37547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 10:29:01.893415 30618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:01.887119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:40469 (local address 127.20.92.62:45207)
0929 10:29:01.887418 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:01.887423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:01.888600 (+ 1177us) server_negotiation.cc:407] Connection header received
0929 10:29:01.890782 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:01.890788 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:01.890844 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:01.890947 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:29:01.892753 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:29:01.892756 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:29:01.892758 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:29:01.892770 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:29:01.892793 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:29:01.892803 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:29:01.892909 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 10:29:01.893183 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 10:29:01.893232 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260929 10:29:01.894821 30900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:01.887567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45207 (local address 127.20.92.2:40469)
0929 10:29:01.888391 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0929 10:29:01.888441 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 10:29:01.890338 (+ 1897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:29:01.891520 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:29:01.891528 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:29:01.891635 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:29:01.892384 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:29:01.892386 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:29:01.892446 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:29:01.892546 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:29:01.892944 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:29:01.892958 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:29:01.892968 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 10:29:01.893201 (+ 233us) client_negotiation.cc:240] Negotiation successful
0929 10:29:01.893523 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":136,"threads_started":1}
I20260929 10:29:01.896668 30898 heartbeater.cc:347] Connected to a master server at 127.20.92.62:45207
I20260929 10:29:01.897040 30898 heartbeater.cc:464] Registering TS with master...
I20260929 10:29:01.897578 30898 heartbeater.cc:511] Master 127.20.92.62:45207 requested a full tablet report, sending...
I20260929 10:29:01.899132 30579 ts_manager.cc:194] Registered new tserver with Master: bfb8727ec6884f2f971d8ac1fbcf3aa0 (127.20.92.2:41913)
W20260929 10:29:02.030937 30901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:29:02.031309 30901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:29:02.031391 30901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:29:02.036100 30901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:29:02.036196 30901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:29:02.036293 30901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:29:02.041781 30901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:37547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:45207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:29:02.043009 30901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:29:02.044449 30901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:29:02.056362 30907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:29:02.056362 30906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:29:02.056547 30901 server_base.cc:1034] running on GCE node
W20260929 10:29:02.056449 30909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:29:02.057096 30901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:29:02.057858 30901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:29:02.059056 30901 hybrid_clock.cc:648] HybridClock initialized: now 1790677742059035 us; error 39 us; skew 500 ppm
I20260929 10:29:02.061658 30901 webserver.cc:460] Webserver started at http://127.20.92.3:44439/ using document root <none> and password file <none>
I20260929 10:29:02.062374 30901 fs_manager.cc:359] Metadata directory not provided
I20260929 10:29:02.062479 30901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:29:02.062734 30901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:29:02.065445 30901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/data/instance:
uuid: "9e5ef8a587a047bba6b403edcbc8e683"
format_stamp: "Formatted at 2026-09-29 10:29:02 on dist-test-slave-9cj5"
server_key: "9e033d444aa7c543c59c130d139b43eb"
server_key_iv: "18089c33489b25b8b1271c486a62c057"
server_key_version: "encryptionkey@0"
I20260929 10:29:02.066131 30901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance:
uuid: "9e5ef8a587a047bba6b403edcbc8e683"
format_stamp: "Formatted at 2026-09-29 10:29:02 on dist-test-slave-9cj5"
server_key: "9e033d444aa7c543c59c130d139b43eb"
server_key_iv: "18089c33489b25b8b1271c486a62c057"
server_key_version: "encryptionkey@0"
I20260929 10:29:02.071026 30901 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 10:29:02.074229 30915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:02.075724 30901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:29:02.075901 30901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "9e5ef8a587a047bba6b403edcbc8e683"
format_stamp: "Formatted at 2026-09-29 10:29:02 on dist-test-slave-9cj5"
server_key: "9e033d444aa7c543c59c130d139b43eb"
server_key_iv: "18089c33489b25b8b1271c486a62c057"
server_key_version: "encryptionkey@0"
I20260929 10:29:02.076040 30901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:29:02.081467 30901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:29:02.082214 30901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:29:02.082422 30901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:29:02.083041 30901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:29:02.084038 30901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:29:02.084111 30901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:02.084183 30901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:29:02.084237 30901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:02.098779 30901 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:35949
I20260929 10:29:02.098802 31028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:35949 every 8 connection(s)
I20260929 10:29:02.099900 30901 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:29:02.103417 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 30901
I20260929 10:29:02.103554 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance
I20260929 10:29:02.103905 20848 external_mini_cluster.cc:1428] Setting key b429176e608def69efb6392739b169c1
I20260929 10:29:02.108093 30618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:02.102075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:43665 (local address 127.20.92.62:45207)
0929 10:29:02.102309 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:02.102314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:02.104082 (+ 1768us) server_negotiation.cc:407] Connection header received
0929 10:29:02.105901 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:02.105906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:02.105981 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:02.106135 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:29:02.107533 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:29:02.107537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:29:02.107539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:29:02.107554 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:29:02.107584 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:29:02.107593 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:29:02.107683 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 10:29:02.107927 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 10:29:02.107978 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260929 10:29:02.109264 31031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:02.102585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45207 (local address 127.20.92.3:43665)
0929 10:29:02.103712 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0929 10:29:02.103779 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 10:29:02.105507 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:29:02.106384 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:29:02.106392 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:29:02.106496 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:29:02.107215 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:29:02.107216 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:29:02.107267 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:29:02.107355 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:29:02.107723 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:29:02.107735 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:29:02.107743 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 10:29:02.107943 (+ 200us) client_negotiation.cc:240] Negotiation successful
0929 10:29:02.108292 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":125,"threads_started":1}
I20260929 10:29:02.110742 31029 heartbeater.cc:347] Connected to a master server at 127.20.92.62:45207
I20260929 10:29:02.111155 31029 heartbeater.cc:464] Registering TS with master...
I20260929 10:29:02.111632 31029 heartbeater.cc:511] Master 127.20.92.62:45207 requested a full tablet report, sending...
I20260929 10:29:02.112845 30579 ts_manager.cc:194] Registered new tserver with Master: 9e5ef8a587a047bba6b403edcbc8e683 (127.20.92.3:35949)
I20260929 10:29:02.119647 20848 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 10:29:02.123344 30618 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:29:02.121865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35632 (local address 127.20.92.62:45207)
0929 10:29:02.122008 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:02.122011 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:02.122403 (+ 392us) server_negotiation.cc:407] Connection header received
0929 10:29:02.122451 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:02.122454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:02.122500 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:02.122576 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:29:02.123055 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35632: BlockingRecv error: recv got EOF from 127.0.0.1:35632 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 10:29:02.123647 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 30639
I20260929 10:29:02.129726 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 30770
I20260929 10:29:02.135401 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 30901
I20260929 10:29:02.141150 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 30548
2026-09-29T10:29:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (939 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:29:02 dist-test-slave-9cj5 krb5kdc[31039](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:29:02 dist-test-slave-9cj5 krb5kdc[31039](info): set up 2 sockets
Sep 29 10:29:02 dist-test-slave-9cj5 krb5kdc[31039](info): commencing operation
krb5kdc: starting...
W20260929 10:29:03.295708 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:29:03 dist-test-slave-9cj5 krb5kdc[31039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:29:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:29:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
I20260929 10:29:03.446126 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:44875
--webserver_interface=127.20.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:38881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:44875
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:29:03.566038 31052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:29:03.566339 31052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:29:03.566381 31052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:29:03.570489 31052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:29:03.570559 31052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:29:03.570578 31052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:29:03.570596 31052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:29:03.570611 31052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:29:03.576001 31052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:38881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:44875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:44875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:29:03.577090 31052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:29:03.578094 31052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:29:03.587422 31060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:29:03.587433 31058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:29:03.587435 31057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:29:03.587877 31052 server_base.cc:1034] running on GCE node
I20260929 10:29:03.588457 31052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:29:03.589635 31052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:29:03.591003 31052 hybrid_clock.cc:648] HybridClock initialized: now 1790677743590970 us; error 46 us; skew 500 ppm
Sep 29 10:29:03 dist-test-slave-9cj5 krb5kdc[31039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677743, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:29:03.594254 31052 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:29:03.595288 31052 webserver.cc:460] Webserver started at http://127.20.92.62:39975/ using document root <none> and password file <none>
I20260929 10:29:03.595881 31052 fs_manager.cc:359] Metadata directory not provided
I20260929 10:29:03.595974 31052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:29:03.596206 31052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:29:03.598428 31052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "b34365c2e46f4fe1b65897715534a4ad"
format_stamp: "Formatted at 2026-09-29 10:29:03 on dist-test-slave-9cj5"
server_key: "c1dc5ed2733c8d09840a775c77ae62f2"
server_key_iv: "e799d58d132fe933ff7184ae92c12840"
server_key_version: "encryptionkey@0"
I20260929 10:29:03.598992 31052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "b34365c2e46f4fe1b65897715534a4ad"
format_stamp: "Formatted at 2026-09-29 10:29:03 on dist-test-slave-9cj5"
server_key: "c1dc5ed2733c8d09840a775c77ae62f2"
server_key_iv: "e799d58d132fe933ff7184ae92c12840"
server_key_version: "encryptionkey@0"
I20260929 10:29:03.602968 31052 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 10:29:03.605908 31067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:03.607415 31052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 10:29:03.607685 31052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "b34365c2e46f4fe1b65897715534a4ad"
format_stamp: "Formatted at 2026-09-29 10:29:03 on dist-test-slave-9cj5"
server_key: "c1dc5ed2733c8d09840a775c77ae62f2"
server_key_iv: "e799d58d132fe933ff7184ae92c12840"
server_key_version: "encryptionkey@0"
I20260929 10:29:03.607846 31052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:29:03.630535 31052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:29:03.634653 31052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:29:03.634913 31052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:29:03.644083 31052 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:44875
I20260929 10:29:03.644507 31119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:44875 every 8 connection(s)
I20260929 10:29:03.645264 31052 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:29:03.648452 31120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:29:03.653669 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 31052
I20260929 10:29:03.653947 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:29:03.654310 20848 external_mini_cluster.cc:1428] Setting key ebf674f85916a723ae205d765d8448d8
I20260929 10:29:03.656404 31120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b34365c2e46f4fe1b65897715534a4ad: Bootstrap starting.
I20260929 10:29:03.658797 31120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b34365c2e46f4fe1b65897715534a4ad: Neither blocks nor log segments found. Creating new log.
I20260929 10:29:03.659574 31120 log.cc:824] T 00000000000000000000000000000000 P b34365c2e46f4fe1b65897715534a4ad: Log is configured to *not* fsync() on all Append() calls
I20260929 10:29:03.661484 31120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b34365c2e46f4fe1b65897715534a4ad: No bootstrap required, opened a new log
Sep 29 10:29:03 dist-test-slave-9cj5 krb5kdc[31039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:29:03.666185 31120 raft_consensus.cc:348] T 00000000000000000000000000000000 P b34365c2e46f4fe1b65897715534a4ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34365c2e46f4fe1b65897715534a4ad" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 44875 } }
I20260929 10:29:03.666451 31120 raft_consensus.cc:374] T 00000000000000000000000000000000 P b34365c2e46f4fe1b65897715534a4ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:29:03.666517 31120 raft_consensus.cc:729] T 00000000000000000000000000000000 P b34365c2e46f4fe1b65897715534a4ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b34365c2e46f4fe1b65897715534a4ad, State: Initialized, Role: FOLLOWER
I20260929 10:29:03.667016 31120 consensus_queue.cc:260] T 00000000000000000000000000000000 P b34365c2e46f4fe1b65897715534a4ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34365c2e46f4fe1b65897715534a4ad" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 44875 } }
I20260929 10:29:03.667188 31120 raft_consensus.cc:388] T 00000000000000000000000000000000 P b34365c2e46f4fe1b65897715534a4ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:29:03.667284 31120 raft_consensus.cc:482] T 00000000000000000000000000000000 P b34365c2e46f4fe1b65897715534a4ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:29:03.667439 31120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b34365c2e46f4fe1b65897715534a4ad [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:29:03.668464 31120 raft_consensus.cc:504] T 00000000000000000000000000000000 P b34365c2e46f4fe1b65897715534a4ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34365c2e46f4fe1b65897715534a4ad" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 44875 } }
I20260929 10:29:03.668852 31120 leader_election.cc:302] T 00000000000000000000000000000000 P b34365c2e46f4fe1b65897715534a4ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b34365c2e46f4fe1b65897715534a4ad; no voters:
I20260929 10:29:03.669252 31120 leader_election.cc:288] T 00000000000000000000000000000000 P b34365c2e46f4fe1b65897715534a4ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:29:03.669720 31126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b34365c2e46f4fe1b65897715534a4ad [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:29:03.669992 31126 raft_consensus.cc:686] T 00000000000000000000000000000000 P b34365c2e46f4fe1b65897715534a4ad [term 1 LEADER]: Becoming Leader. State: Replica: b34365c2e46f4fe1b65897715534a4ad, State: Running, Role: LEADER
I20260929 10:29:03.670616 31123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:03.656099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53078 (local address 127.20.92.62:44875)
0929 10:29:03.656611 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:03.656628 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:03.656666 (+ 38us) server_negotiation.cc:407] Connection header received
0929 10:29:03.657368 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:03.657386 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:03.657734 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:03.658161 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:29:03.658977 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:03.660046 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:03.660913 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:03.661414 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:03.661587 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:03.664473 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:29:03.664479 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:29:03.664493 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:29:03.664522 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:29:03.666839 (+ 2317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:29:03.667446 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:29:03.667453 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:29:03.667459 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:29:03.667644 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:29:03.667973 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:29:03.667976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:29:03.667978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:29:03.668426 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:29:03.668668 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 10:29:03.669001 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 10:29:03.669342 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":152,"threads_started":1}
I20260929 10:29:03.670728 31120 sys_catalog.cc:564] T 00000000000000000000000000000000 P b34365c2e46f4fe1b65897715534a4ad [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:29:03.671160 31126 consensus_queue.cc:237] T 00000000000000000000000000000000 P b34365c2e46f4fe1b65897715534a4ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34365c2e46f4fe1b65897715534a4ad" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 44875 } }
I20260929 10:29:03.672564 31127 sys_catalog.cc:455] T 00000000000000000000000000000000 P b34365c2e46f4fe1b65897715534a4ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader b34365c2e46f4fe1b65897715534a4ad. Latest consensus state: current_term: 1 leader_uuid: "b34365c2e46f4fe1b65897715534a4ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34365c2e46f4fe1b65897715534a4ad" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 44875 } } }
I20260929 10:29:03.672698 31127 sys_catalog.cc:458] T 00000000000000000000000000000000 P b34365c2e46f4fe1b65897715534a4ad [sys.catalog]: This master's current role is: LEADER
I20260929 10:29:03.672991 31125 sys_catalog.cc:455] T 00000000000000000000000000000000 P b34365c2e46f4fe1b65897715534a4ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b34365c2e46f4fe1b65897715534a4ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34365c2e46f4fe1b65897715534a4ad" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 44875 } } }
I20260929 10:29:03.673113 31125 sys_catalog.cc:458] T 00000000000000000000000000000000 P b34365c2e46f4fe1b65897715534a4ad [sys.catalog]: This master's current role is: LEADER
I20260929 10:29:03.673269 31134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:29:03.677088 31134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:29:03.683684 31134 catalog_manager.cc:1347] Generated new cluster ID: 473b9de9517c415b8df16fdac8dbb287
I20260929 10:29:03.683781 31134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:29:03.705139 31134 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:29:03.706588 31134 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:29:03.720665 31134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b34365c2e46f4fe1b65897715534a4ad: Generated new TSK 0
I20260929 10:29:03.721594 31134 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
I20260929 10:29:03.758313 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:0
--local_ip_for_outbound_sockets=127.20.92.1
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:44875
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:38881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:29:03.881372 31146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:29:03.881726 31146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:29:03.881837 31146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:29:03.886812 31146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:29:03.886945 31146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:29:03.887053 31146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:29:03.892885 31146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:38881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:44875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:29:03.894146 31146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:29:03.895292 31146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:29:03.906293 31151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:29:03.906232 31154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:29:03.906419 31146 server_base.cc:1034] running on GCE node
W20260929 10:29:03.906665 31152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:29:03.907222 31146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:29:03.907902 31146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:29:03.909140 31146 hybrid_clock.cc:648] HybridClock initialized: now 1790677743909104 us; error 56 us; skew 500 ppm
Sep 29 10:29:03 dist-test-slave-9cj5 krb5kdc[31039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677743, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:29:03.912477 31146 init.cc:377] Logged in from keytab as kudu/127.20.92.1@KRBTEST.COM (short username kudu)
I20260929 10:29:03.913509 31146 webserver.cc:460] Webserver started at http://127.20.92.1:34517/ using document root <none> and password file <none>
I20260929 10:29:03.914166 31146 fs_manager.cc:359] Metadata directory not provided
I20260929 10:29:03.914263 31146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:29:03.914492 31146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:29:03.916558 31146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/data/instance:
uuid: "65582f059e8142f3ae495d608f61f6a7"
format_stamp: "Formatted at 2026-09-29 10:29:03 on dist-test-slave-9cj5"
server_key: "ce4be01c53504750b12e724a088559a3"
server_key_iv: "bba4a41447f67664344bae74a4f13e21"
server_key_version: "encryptionkey@0"
I20260929 10:29:03.917138 31146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance:
uuid: "65582f059e8142f3ae495d608f61f6a7"
format_stamp: "Formatted at 2026-09-29 10:29:03 on dist-test-slave-9cj5"
server_key: "ce4be01c53504750b12e724a088559a3"
server_key_iv: "bba4a41447f67664344bae74a4f13e21"
server_key_version: "encryptionkey@0"
I20260929 10:29:03.921154 31146 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 10:29:03.923936 31161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:03.925371 31146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:29:03.925542 31146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "65582f059e8142f3ae495d608f61f6a7"
format_stamp: "Formatted at 2026-09-29 10:29:03 on dist-test-slave-9cj5"
server_key: "ce4be01c53504750b12e724a088559a3"
server_key_iv: "bba4a41447f67664344bae74a4f13e21"
server_key_version: "encryptionkey@0"
I20260929 10:29:03.925681 31146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:29:03.941646 31146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:29:03.945165 31146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:29:03.945398 31146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:29:03.946094 31146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:29:03.947006 31146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:29:03.947081 31146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:03.947155 31146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:29:03.947211 31146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:03.957389 31146 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:33083
I20260929 10:29:03.957479 31274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:33083 every 8 connection(s)
I20260929 10:29:03.958554 31146 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
I20260929 10:29:03.966048 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 31146
I20260929 10:29:03.966167 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance
I20260929 10:29:03.966459 20848 external_mini_cluster.cc:1428] Setting key e461ca36797a6d7a9b04586022af7389
Sep 29 10:29:03 dist-test-slave-9cj5 krb5kdc[31039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677743, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:29:03.974614 31123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:03.960816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:38719 (local address 127.20.92.62:44875)
0929 10:29:03.961146 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:03.961150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:03.961920 (+ 770us) server_negotiation.cc:407] Connection header received
0929 10:29:03.963786 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:03.963789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:03.963853 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:03.963966 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:29:03.965262 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:03.965902 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:03.967468 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:03.967752 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:03.967864 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:03.970605 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:29:03.970609 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:29:03.970614 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:29:03.970645 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:29:03.972285 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:29:03.972928 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:29:03.972932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:29:03.972934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:29:03.973016 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:29:03.973380 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:29:03.973385 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:29:03.973386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:29:03.973593 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:29:03.973776 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 10:29:03.974311 (+ 535us) server_negotiation.cc:299] Negotiation successful
0929 10:29:03.974446 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233}
I20260929 10:29:03.975790 31277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:03.961107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:44875 (local address 127.20.92.1:38719)
0929 10:29:03.961668 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0929 10:29:03.961719 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 10:29:03.963524 (+ 1805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:29:03.964157 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:29:03.964165 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:29:03.964568 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:29:03.965082 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:03.965106 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:03.966164 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:03.966171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:03.967305 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:03.967317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:03.967954 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:03.967958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:03.968162 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:03.968823 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:29:03.968840 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:29:03.970402 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:29:03.972406 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:29:03.972412 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:29:03.972432 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:29:03.972823 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:29:03.973122 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:29:03.973125 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:29:03.973127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:29:03.973276 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:29:03.973773 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:29:03.973781 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:29:03.974060 (+ 279us) client_negotiation.cc:769] Sending connection context
0929 10:29:03.974259 (+ 199us) client_negotiation.cc:240] Negotiation successful
0929 10:29:03.974634 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":139,"threads_started":1}
WARNING: no policy specified for kudu/127.20.92.2@KRBTEST.COM; defaulting to no policy
I20260929 10:29:03.977421 31275 heartbeater.cc:347] Connected to a master server at 127.20.92.62:44875
I20260929 10:29:03.977631 31275 heartbeater.cc:464] Registering TS with master...
I20260929 10:29:03.978247 31275 heartbeater.cc:511] Master 127.20.92.62:44875 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.2@KRBTEST.COM" created.
I20260929 10:29:03.980049 31078 ts_manager.cc:194] Registered new tserver with Master: 65582f059e8142f3ae495d608f61f6a7 (127.20.92.1:33083)
I20260929 10:29:03.981701 31078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.1@KRBTEST.COM'} at 127.20.92.1:38719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
I20260929 10:29:04.002744 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:0
--local_ip_for_outbound_sockets=127.20.92.2
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:44875
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:38881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:29:04.116813 31280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:29:04.117087 31280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:29:04.117131 31280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:29:04.121253 31280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:29:04.121335 31280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:29:04.121397 31280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:29:04.127431 31280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:38881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:44875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:29:04.128587 31280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:29:04.129952 31280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:29:04.140416 31288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:29:04.140416 31285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:29:04.140832 31286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:29:04.140882 31280 server_base.cc:1034] running on GCE node
I20260929 10:29:04.141355 31280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:29:04.142100 31280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:29:04.143306 31280 hybrid_clock.cc:648] HybridClock initialized: now 1790677744143269 us; error 49 us; skew 500 ppm
Sep 29 10:29:04 dist-test-slave-9cj5 krb5kdc[31039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677744, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:29:04.146677 31280 init.cc:377] Logged in from keytab as kudu/127.20.92.2@KRBTEST.COM (short username kudu)
I20260929 10:29:04.147720 31280 webserver.cc:460] Webserver started at http://127.20.92.2:36031/ using document root <none> and password file <none>
I20260929 10:29:04.148345 31280 fs_manager.cc:359] Metadata directory not provided
I20260929 10:29:04.148452 31280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:29:04.148703 31280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:29:04.150995 31280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/data/instance:
uuid: "76427dd6d6a64aa486fd5dafa71c087a"
format_stamp: "Formatted at 2026-09-29 10:29:04 on dist-test-slave-9cj5"
server_key: "ad9fec8ccafc36e4747d9de1657d18dc"
server_key_iv: "85fe8928ae4342c287f8f8725a0f91da"
server_key_version: "encryptionkey@0"
I20260929 10:29:04.151608 31280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance:
uuid: "76427dd6d6a64aa486fd5dafa71c087a"
format_stamp: "Formatted at 2026-09-29 10:29:04 on dist-test-slave-9cj5"
server_key: "ad9fec8ccafc36e4747d9de1657d18dc"
server_key_iv: "85fe8928ae4342c287f8f8725a0f91da"
server_key_version: "encryptionkey@0"
I20260929 10:29:04.155896 31280 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:29:04.158854 31295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:04.160426 31280 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:29:04.160598 31280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "76427dd6d6a64aa486fd5dafa71c087a"
format_stamp: "Formatted at 2026-09-29 10:29:04 on dist-test-slave-9cj5"
server_key: "ad9fec8ccafc36e4747d9de1657d18dc"
server_key_iv: "85fe8928ae4342c287f8f8725a0f91da"
server_key_version: "encryptionkey@0"
I20260929 10:29:04.160741 31280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:29:04.195533 31280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:29:04.199349 31280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:29:04.199586 31280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:29:04.200228 31280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:29:04.201154 31280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:29:04.201228 31280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:04.201301 31280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:29:04.201352 31280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:04.211910 31280 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:46877
I20260929 10:29:04.211973 31408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:46877 every 8 connection(s)
I20260929 10:29:04.212957 31280 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
I20260929 10:29:04.220732 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 31280
I20260929 10:29:04.220925 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance
I20260929 10:29:04.221236 20848 external_mini_cluster.cc:1428] Setting key 87b5c6a6e0d61cce5e57b7cb4f5732f6
Sep 29 10:29:04 dist-test-slave-9cj5 krb5kdc[31039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677744, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:29:04.232329 31123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:04.215401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:53375 (local address 127.20.92.62:44875)
0929 10:29:04.215581 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:04.215586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:04.216483 (+ 897us) server_negotiation.cc:407] Connection header received
0929 10:29:04.218605 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:04.218608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:04.218664 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:04.218759 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:29:04.220050 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:04.220619 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:04.222396 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:04.222748 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:04.222874 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:04.226606 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:29:04.226615 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:29:04.226622 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:29:04.226666 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:29:04.228978 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:29:04.230062 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:29:04.230070 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:29:04.230075 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:29:04.230200 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:29:04.230765 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:29:04.230768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:29:04.230770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:29:04.231191 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:29:04.231332 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 10:29:04.231957 (+ 625us) server_negotiation.cc:299] Negotiation successful
0929 10:29:04.232116 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
WARNING: no policy specified for kudu/127.20.92.3@KRBTEST.COM; defaulting to no policy
I20260929 10:29:04.233441 31411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:04.215733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:44875 (local address 127.20.92.2:53375)
0929 10:29:04.216292 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0929 10:29:04.216342 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 10:29:04.218334 (+ 1992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:29:04.218971 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:29:04.218979 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:29:04.219375 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:29:04.219895 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:04.219910 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:04.220815 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:04.220820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:04.222245 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:04.222256 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:04.222961 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:04.222965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:04.223178 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:04.223856 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:29:04.223884 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:29:04.226350 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:29:04.229478 (+ 3128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:29:04.229485 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:29:04.229502 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:29:04.229909 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:29:04.230343 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:29:04.230346 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:29:04.230347 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:29:04.230603 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:29:04.231359 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:29:04.231366 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:29:04.231643 (+ 277us) client_negotiation.cc:769] Sending connection context
0929 10:29:04.231824 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 10:29:04.232138 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":144,"threads_started":1}
I20260929 10:29:04.235168 31409 heartbeater.cc:347] Connected to a master server at 127.20.92.62:44875
I20260929 10:29:04.235430 31409 heartbeater.cc:464] Registering TS with master...
I20260929 10:29:04.235985 31409 heartbeater.cc:511] Master 127.20.92.62:44875 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.3@KRBTEST.COM" created.
I20260929 10:29:04.237499 31078 ts_manager.cc:194] Registered new tserver with Master: 76427dd6d6a64aa486fd5dafa71c087a (127.20.92.2:46877)
I20260929 10:29:04.238257 31078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.2@KRBTEST.COM'} at 127.20.92.2:53375
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
I20260929 10:29:04.259476 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:0
--local_ip_for_outbound_sockets=127.20.92.3
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:44875
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:38881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:29:04.374867 31414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:29:04.375167 31414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:29:04.375213 31414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:29:04.380373 31414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:29:04.380474 31414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:29:04.380582 31414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:29:04.387076 31414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:38881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:44875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:29:04.388440 31414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:29:04.389652 31414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:29:04.400338 31419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:29:04.400338 31420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:29:04.400457 31414 server_base.cc:1034] running on GCE node
W20260929 10:29:04.400338 31422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:29:04.401129 31414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:29:04.401824 31414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:29:04.402989 31414 hybrid_clock.cc:648] HybridClock initialized: now 1790677744402956 us; error 53 us; skew 500 ppm
Sep 29 10:29:04 dist-test-slave-9cj5 krb5kdc[31039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677744, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:29:04.406348 31414 init.cc:377] Logged in from keytab as kudu/127.20.92.3@KRBTEST.COM (short username kudu)
I20260929 10:29:04.407472 31414 webserver.cc:460] Webserver started at http://127.20.92.3:40509/ using document root <none> and password file <none>
I20260929 10:29:04.408097 31414 fs_manager.cc:359] Metadata directory not provided
I20260929 10:29:04.408169 31414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:29:04.408393 31414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:29:04.410682 31414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/data/instance:
uuid: "690a0db9ce0d402985fd615d3d879679"
format_stamp: "Formatted at 2026-09-29 10:29:04 on dist-test-slave-9cj5"
server_key: "71983ced5bbd406171afbbad0ab3ce63"
server_key_iv: "934ba242186ffe5df95cf138e37e5f75"
server_key_version: "encryptionkey@0"
I20260929 10:29:04.411309 31414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance:
uuid: "690a0db9ce0d402985fd615d3d879679"
format_stamp: "Formatted at 2026-09-29 10:29:04 on dist-test-slave-9cj5"
server_key: "71983ced5bbd406171afbbad0ab3ce63"
server_key_iv: "934ba242186ffe5df95cf138e37e5f75"
server_key_version: "encryptionkey@0"
I20260929 10:29:04.415329 31414 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 10:29:04.418406 31429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:04.419835 31414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:29:04.419996 31414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "690a0db9ce0d402985fd615d3d879679"
format_stamp: "Formatted at 2026-09-29 10:29:04 on dist-test-slave-9cj5"
server_key: "71983ced5bbd406171afbbad0ab3ce63"
server_key_iv: "934ba242186ffe5df95cf138e37e5f75"
server_key_version: "encryptionkey@0"
I20260929 10:29:04.420127 31414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:29:04.440470 31414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:29:04.444619 31414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:29:04.444888 31414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:29:04.445632 31414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:29:04.446597 31414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:29:04.446650 31414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:04.446694 31414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:29:04.446712 31414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:04.457027 31414 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:44107
I20260929 10:29:04.457122 31542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:44107 every 8 connection(s)
I20260929 10:29:04.458104 31414 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:29:04.467222 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 31414
I20260929 10:29:04.467399 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance
I20260929 10:29:04.467883 20848 external_mini_cluster.cc:1428] Setting key 5bb216c771976a4b5b8591872099e449
Sep 29 10:29:04 dist-test-slave-9cj5 krb5kdc[31039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677744, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:29:04.474448 31123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:04.460110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:54737 (local address 127.20.92.62:44875)
0929 10:29:04.460301 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:04.460306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:04.461182 (+ 876us) server_negotiation.cc:407] Connection header received
0929 10:29:04.462715 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:04.462719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:04.462779 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:04.462896 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:29:04.464244 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:04.464867 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:04.466521 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:04.466801 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:04.466939 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:04.470282 (+ 3343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:29:04.470287 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:29:04.470290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:29:04.470322 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:29:04.471996 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:29:04.472576 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:29:04.472580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:29:04.472581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:29:04.472674 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:29:04.473091 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:29:04.473095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:29:04.473096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:29:04.473342 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:29:04.473459 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 10:29:04.474195 (+ 736us) server_negotiation.cc:299] Negotiation successful
0929 10:29:04.474320 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 10:29:04.475258 31545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:04.460464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:44875 (local address 127.20.92.3:54737)
0929 10:29:04.460955 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 10:29:04.461000 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 10:29:04.462461 (+ 1461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:29:04.463076 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:29:04.463084 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:29:04.463466 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:29:04.464090 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:04.464107 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:04.465005 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:04.465008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:04.466364 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:04.466375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:04.466986 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:04.466990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:04.467185 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:04.467940 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:29:04.467958 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:29:04.470108 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:29:04.472134 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:29:04.472141 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:29:04.472160 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:29:04.472457 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:29:04.472785 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:29:04.472788 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:29:04.472790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:29:04.472986 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:29:04.473456 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:29:04.473464 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:29:04.473799 (+ 335us) client_negotiation.cc:769] Sending connection context
0929 10:29:04.474023 (+ 224us) client_negotiation.cc:240] Negotiation successful
0929 10:29:04.474343 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":123,"threads_started":1}
I20260929 10:29:04.476543 31543 heartbeater.cc:347] Connected to a master server at 127.20.92.62:44875
I20260929 10:29:04.476814 31543 heartbeater.cc:464] Registering TS with master...
I20260929 10:29:04.477345 31543 heartbeater.cc:511] Master 127.20.92.62:44875 requested a full tablet report, sending...
I20260929 10:29:04.478600 31078 ts_manager.cc:194] Registered new tserver with Master: 690a0db9ce0d402985fd615d3d879679 (127.20.92.3:44107)
I20260929 10:29:04.479264 31078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.3@KRBTEST.COM'} at 127.20.92.3:54737
I20260929 10:29:04.482609 20848 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:29:04.493263 31123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:04.484605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53092 (local address 127.20.92.62:44875)
0929 10:29:04.484875 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:04.484879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:04.484975 (+ 96us) server_negotiation.cc:407] Connection header received
0929 10:29:04.485069 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:04.485071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:04.485112 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:04.485216 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:29:04.486153 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:04.486675 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:04.487607 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:04.487884 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:04.487982 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:04.489598 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:29:04.489601 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:29:04.489603 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:29:04.489628 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:29:04.491252 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:29:04.491752 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:29:04.491755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:29:04.491756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:29:04.491842 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:29:04.492192 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:29:04.492195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:29:04.492196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:29:04.492456 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:29:04.492568 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 10:29:04.492873 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 10:29:04.493002 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260929 10:29:04.497246 31078 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:29:04.500716 31078 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:29:04.517787 31555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:04.511374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:33083 (local address 127.0.0.1:39094)
0929 10:29:04.512185 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0929 10:29:04.512199 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:29:04.512282 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:29:04.513592 (+ 1310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:29:04.513595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:29:04.513620 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:29:04.513720 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:04.513727 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:04.515133 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:04.515137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:04.516209 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:04.516221 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:04.517183 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:04.517187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:04.517295 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:04.517338 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 10:29:04.517473 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 10:29:04.517605 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":40,"thread_start_us":111,"threads_started":1}
I20260929 10:29:04.517798 31557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:04.512130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39094 (local address 127.20.92.1:33083)
0929 10:29:04.512536 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:04.512548 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:04.512580 (+ 32us) server_negotiation.cc:407] Connection header received
0929 10:29:04.512677 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:04.512684 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:04.512993 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:04.513225 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:29:04.513883 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:04.514997 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:04.516357 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:04.517025 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:04.517153 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:04.517309 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 10:29:04.517504 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 10:29:04.517605 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":84,"threads_started":1}
I20260929 10:29:04.518164 31554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:04.511195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:44107 (local address 127.0.0.1:48652)
0929 10:29:04.511739 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0929 10:29:04.511809 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 10:29:04.511941 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:29:04.513222 (+ 1281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:29:04.513228 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:29:04.513252 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:29:04.513412 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:04.513428 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:04.514867 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:04.514872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:04.516264 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:04.516276 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:04.517577 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:04.517582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:04.517720 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:04.517793 (+ 73us) client_negotiation.cc:769] Sending connection context
0929 10:29:04.517868 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 10:29:04.517952 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":118,"thread_start_us":103,"threads_started":1}
I20260929 10:29:04.519050 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:04.511421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48652 (local address 127.20.92.3:44107)
0929 10:29:04.512608 (+ 1187us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:04.512621 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:04.512644 (+ 23us) server_negotiation.cc:407] Connection header received
0929 10:29:04.512719 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:04.512724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:04.513061 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:04.513224 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:29:04.513563 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:04.514686 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:04.516474 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:04.517408 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:04.518466 (+ 1058us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:04.518647 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 10:29:04.518709 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 10:29:04.518834 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1075,"thread_start_us":162,"threads_started":1}
I20260929 10:29:04.519402 31558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:04.512485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:46877 (local address 127.0.0.1:45412)
0929 10:29:04.512912 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0929 10:29:04.512929 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:29:04.513028 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:29:04.513517 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:29:04.513520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:29:04.513538 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:29:04.513638 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:04.513644 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:04.516349 (+ 2705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:04.516352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:04.517712 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:04.517723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:04.519002 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:04.519006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:04.519095 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:04.519118 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 10:29:04.519191 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 10:29:04.519262 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":18,"thread_start_us":120,"threads_started":1}
I20260929 10:29:04.519527 31559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:04.512645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45412 (local address 127.20.92.2:46877)
0929 10:29:04.513068 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:04.513080 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:04.513102 (+ 22us) server_negotiation.cc:407] Connection header received
0929 10:29:04.513188 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:04.513192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:04.513368 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:04.513537 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:29:04.513776 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:04.515283 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:04.517905 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:04.518849 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:04.519005 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:04.519173 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 10:29:04.519239 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 10:29:04.519350 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":140,"threads_started":1}
I20260929 10:29:04.521378 31209 tablet_service.cc:1467] Processing CreateTablet for tablet eba78eb6d09d49ed9c364265e896e453 (DEFAULT_TABLE table=test-table [id=cdd61829738d4b23a462065ae1a54396]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:29:04.522791 31209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eba78eb6d09d49ed9c364265e896e453. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:29:04.522464 31477 tablet_service.cc:1467] Processing CreateTablet for tablet eba78eb6d09d49ed9c364265e896e453 (DEFAULT_TABLE table=test-table [id=cdd61829738d4b23a462065ae1a54396]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:29:04.522462 31343 tablet_service.cc:1467] Processing CreateTablet for tablet eba78eb6d09d49ed9c364265e896e453 (DEFAULT_TABLE table=test-table [id=cdd61829738d4b23a462065ae1a54396]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:29:04.523497 31343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eba78eb6d09d49ed9c364265e896e453. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:29:04.523523 31477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eba78eb6d09d49ed9c364265e896e453. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:29:04.530102 31560 tablet_bootstrap.cc:492] T eba78eb6d09d49ed9c364265e896e453 P 65582f059e8142f3ae495d608f61f6a7: Bootstrap starting.
I20260929 10:29:04.532025 31562 tablet_bootstrap.cc:492] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679: Bootstrap starting.
I20260929 10:29:04.532025 31561 tablet_bootstrap.cc:492] T eba78eb6d09d49ed9c364265e896e453 P 76427dd6d6a64aa486fd5dafa71c087a: Bootstrap starting.
I20260929 10:29:04.532814 31560 tablet_bootstrap.cc:654] T eba78eb6d09d49ed9c364265e896e453 P 65582f059e8142f3ae495d608f61f6a7: Neither blocks nor log segments found. Creating new log.
I20260929 10:29:04.533845 31560 log.cc:824] T eba78eb6d09d49ed9c364265e896e453 P 65582f059e8142f3ae495d608f61f6a7: Log is configured to *not* fsync() on all Append() calls
I20260929 10:29:04.534343 31562 tablet_bootstrap.cc:654] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679: Neither blocks nor log segments found. Creating new log.
I20260929 10:29:04.534858 31561 tablet_bootstrap.cc:654] T eba78eb6d09d49ed9c364265e896e453 P 76427dd6d6a64aa486fd5dafa71c087a: Neither blocks nor log segments found. Creating new log.
I20260929 10:29:04.535835 31560 tablet_bootstrap.cc:492] T eba78eb6d09d49ed9c364265e896e453 P 65582f059e8142f3ae495d608f61f6a7: No bootstrap required, opened a new log
I20260929 10:29:04.535954 31562 log.cc:824] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679: Log is configured to *not* fsync() on all Append() calls
I20260929 10:29:04.535954 31561 log.cc:824] T eba78eb6d09d49ed9c364265e896e453 P 76427dd6d6a64aa486fd5dafa71c087a: Log is configured to *not* fsync() on all Append() calls
I20260929 10:29:04.536077 31560 ts_tablet_manager.cc:1397] T eba78eb6d09d49ed9c364265e896e453 P 65582f059e8142f3ae495d608f61f6a7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 10:29:04.538317 31562 tablet_bootstrap.cc:492] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679: No bootstrap required, opened a new log
I20260929 10:29:04.538419 31561 tablet_bootstrap.cc:492] T eba78eb6d09d49ed9c364265e896e453 P 76427dd6d6a64aa486fd5dafa71c087a: No bootstrap required, opened a new log
I20260929 10:29:04.538570 31562 ts_tablet_manager.cc:1397] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 10:29:04.538659 31561 ts_tablet_manager.cc:1397] T eba78eb6d09d49ed9c364265e896e453 P 76427dd6d6a64aa486fd5dafa71c087a: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 10:29:04.541087 31560 raft_consensus.cc:348] T eba78eb6d09d49ed9c364265e896e453 P 65582f059e8142f3ae495d608f61f6a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "690a0db9ce0d402985fd615d3d879679" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 44107 } } peers { permanent_uuid: "65582f059e8142f3ae495d608f61f6a7" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 33083 } } peers { permanent_uuid: "76427dd6d6a64aa486fd5dafa71c087a" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 46877 } }
I20260929 10:29:04.541419 31560 raft_consensus.cc:374] T eba78eb6d09d49ed9c364265e896e453 P 65582f059e8142f3ae495d608f61f6a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:29:04.541482 31560 raft_consensus.cc:729] T eba78eb6d09d49ed9c364265e896e453 P 65582f059e8142f3ae495d608f61f6a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65582f059e8142f3ae495d608f61f6a7, State: Initialized, Role: FOLLOWER
I20260929 10:29:04.542011 31560 consensus_queue.cc:260] T eba78eb6d09d49ed9c364265e896e453 P 65582f059e8142f3ae495d608f61f6a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "690a0db9ce0d402985fd615d3d879679" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 44107 } } peers { permanent_uuid: "65582f059e8142f3ae495d608f61f6a7" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 33083 } } peers { permanent_uuid: "76427dd6d6a64aa486fd5dafa71c087a" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 46877 } }
I20260929 10:29:04.542582 31561 raft_consensus.cc:348] T eba78eb6d09d49ed9c364265e896e453 P 76427dd6d6a64aa486fd5dafa71c087a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "690a0db9ce0d402985fd615d3d879679" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 44107 } } peers { permanent_uuid: "65582f059e8142f3ae495d608f61f6a7" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 33083 } } peers { permanent_uuid: "76427dd6d6a64aa486fd5dafa71c087a" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 46877 } }
I20260929 10:29:04.542850 31561 raft_consensus.cc:374] T eba78eb6d09d49ed9c364265e896e453 P 76427dd6d6a64aa486fd5dafa71c087a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:29:04.542665 31562 raft_consensus.cc:348] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "690a0db9ce0d402985fd615d3d879679" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 44107 } } peers { permanent_uuid: "65582f059e8142f3ae495d608f61f6a7" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 33083 } } peers { permanent_uuid: "76427dd6d6a64aa486fd5dafa71c087a" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 46877 } }
I20260929 10:29:04.542898 31561 raft_consensus.cc:729] T eba78eb6d09d49ed9c364265e896e453 P 76427dd6d6a64aa486fd5dafa71c087a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76427dd6d6a64aa486fd5dafa71c087a, State: Initialized, Role: FOLLOWER
I20260929 10:29:04.542944 31562 raft_consensus.cc:374] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:29:04.542995 31562 raft_consensus.cc:729] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 690a0db9ce0d402985fd615d3d879679, State: Initialized, Role: FOLLOWER
I20260929 10:29:04.543299 31560 ts_tablet_manager.cc:1428] T eba78eb6d09d49ed9c364265e896e453 P 65582f059e8142f3ae495d608f61f6a7: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260929 10:29:04.543335 31561 consensus_queue.cc:260] T eba78eb6d09d49ed9c364265e896e453 P 76427dd6d6a64aa486fd5dafa71c087a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "690a0db9ce0d402985fd615d3d879679" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 44107 } } peers { permanent_uuid: "65582f059e8142f3ae495d608f61f6a7" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 33083 } } peers { permanent_uuid: "76427dd6d6a64aa486fd5dafa71c087a" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 46877 } }
I20260929 10:29:04.543470 31562 consensus_queue.cc:260] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "690a0db9ce0d402985fd615d3d879679" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 44107 } } peers { permanent_uuid: "65582f059e8142f3ae495d608f61f6a7" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 33083 } } peers { permanent_uuid: "76427dd6d6a64aa486fd5dafa71c087a" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 46877 } }
I20260929 10:29:04.543677 31275 heartbeater.cc:503] Master 127.20.92.62:44875 was elected leader, sending a full tablet report...
I20260929 10:29:04.544226 31543 heartbeater.cc:503] Master 127.20.92.62:44875 was elected leader, sending a full tablet report...
I20260929 10:29:04.544269 31561 ts_tablet_manager.cc:1428] T eba78eb6d09d49ed9c364265e896e453 P 76427dd6d6a64aa486fd5dafa71c087a: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260929 10:29:04.544569 31562 ts_tablet_manager.cc:1428] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 10:29:04.544751 31409 heartbeater.cc:503] Master 127.20.92.62:44875 was elected leader, sending a full tablet report...
I20260929 10:29:04.587754 31568 raft_consensus.cc:482] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:29:04.588058 31568 raft_consensus.cc:504] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "690a0db9ce0d402985fd615d3d879679" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 44107 } } peers { permanent_uuid: "65582f059e8142f3ae495d608f61f6a7" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 33083 } } peers { permanent_uuid: "76427dd6d6a64aa486fd5dafa71c087a" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 46877 } }
I20260929 10:29:04.589522 31568 leader_election.cc:288] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65582f059e8142f3ae495d608f61f6a7 (127.20.92.1:33083), 76427dd6d6a64aa486fd5dafa71c087a (127.20.92.2:46877)
I20260929 10:29:04.593818 31559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:04.589981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:36271 (local address 127.20.92.2:46877)
0929 10:29:04.590128 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:04.590133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:04.590366 (+ 233us) server_negotiation.cc:407] Connection header received
0929 10:29:04.590429 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:04.590432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:04.590484 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:04.590574 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:29:04.590878 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:04.591485 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:04.592660 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:04.593117 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:04.593224 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:04.593315 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 10:29:04.593553 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 10:29:04.593643 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 10:29:04.593809 31569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:04.589725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:46877 (local address 127.20.92.3:36271)
0929 10:29:04.590162 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0929 10:29:04.590175 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:29:04.590289 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:29:04.590593 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:29:04.590595 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:29:04.590632 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:29:04.590750 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:04.590756 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:04.591624 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:04.591626 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:04.592470 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:04.592479 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:04.593293 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:04.593297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:04.593401 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:04.593433 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 10:29:04.593509 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 10:29:04.593604 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":125,"threads_started":1}
I20260929 10:29:04.593858 31557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:04.589880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:55855 (local address 127.20.92.1:33083)
0929 10:29:04.590015 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:04.590020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:04.590038 (+ 18us) server_negotiation.cc:407] Connection header received
0929 10:29:04.590157 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:04.590160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:04.590224 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:04.590304 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:29:04.590662 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:04.591358 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:04.592605 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:04.593148 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:04.593326 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:04.593399 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 10:29:04.593613 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 10:29:04.593704 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 10:29:04.593818 31545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:04.589699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:33083 (local address 127.20.92.3:55855)
0929 10:29:04.589927 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 10:29:04.589944 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:29:04.590028 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:29:04.590367 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:29:04.590370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:29:04.590394 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:29:04.590545 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:04.590555 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:04.591487 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:04.591491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:04.592451 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:04.592461 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:04.593346 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:04.593350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:04.593459 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:04.593477 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 10:29:04.593583 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 10:29:04.593655 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":1}
I20260929 10:29:04.594625 31229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba78eb6d09d49ed9c364265e896e453" candidate_uuid: "690a0db9ce0d402985fd615d3d879679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65582f059e8142f3ae495d608f61f6a7" is_pre_election: true
I20260929 10:29:04.594625 31363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba78eb6d09d49ed9c364265e896e453" candidate_uuid: "690a0db9ce0d402985fd615d3d879679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76427dd6d6a64aa486fd5dafa71c087a" is_pre_election: true
I20260929 10:29:04.594978 31229 raft_consensus.cc:2454] T eba78eb6d09d49ed9c364265e896e453 P 65582f059e8142f3ae495d608f61f6a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 690a0db9ce0d402985fd615d3d879679 in term 0.
I20260929 10:29:04.594978 31363 raft_consensus.cc:2454] T eba78eb6d09d49ed9c364265e896e453 P 76427dd6d6a64aa486fd5dafa71c087a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 690a0db9ce0d402985fd615d3d879679 in term 0.
I20260929 10:29:04.595495 31433 leader_election.cc:302] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 690a0db9ce0d402985fd615d3d879679, 76427dd6d6a64aa486fd5dafa71c087a; no voters:
I20260929 10:29:04.595794 31568 raft_consensus.cc:2781] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:29:04.595897 31568 raft_consensus.cc:482] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:29:04.595949 31568 raft_consensus.cc:3037] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:29:04.597035 31568 raft_consensus.cc:504] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "690a0db9ce0d402985fd615d3d879679" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 44107 } } peers { permanent_uuid: "65582f059e8142f3ae495d608f61f6a7" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 33083 } } peers { permanent_uuid: "76427dd6d6a64aa486fd5dafa71c087a" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 46877 } }
I20260929 10:29:04.597409 31568 leader_election.cc:288] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679 [CANDIDATE]: Term 1 election: Requested vote from peers 65582f059e8142f3ae495d608f61f6a7 (127.20.92.1:33083), 76427dd6d6a64aa486fd5dafa71c087a (127.20.92.2:46877)
I20260929 10:29:04.597919 31363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba78eb6d09d49ed9c364265e896e453" candidate_uuid: "690a0db9ce0d402985fd615d3d879679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76427dd6d6a64aa486fd5dafa71c087a"
I20260929 10:29:04.597913 31229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba78eb6d09d49ed9c364265e896e453" candidate_uuid: "690a0db9ce0d402985fd615d3d879679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65582f059e8142f3ae495d608f61f6a7"
I20260929 10:29:04.598055 31363 raft_consensus.cc:3037] T eba78eb6d09d49ed9c364265e896e453 P 76427dd6d6a64aa486fd5dafa71c087a [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:29:04.598055 31229 raft_consensus.cc:3037] T eba78eb6d09d49ed9c364265e896e453 P 65582f059e8142f3ae495d608f61f6a7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:29:04.599386 31363 raft_consensus.cc:2454] T eba78eb6d09d49ed9c364265e896e453 P 76427dd6d6a64aa486fd5dafa71c087a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 690a0db9ce0d402985fd615d3d879679 in term 1.
I20260929 10:29:04.599386 31229 raft_consensus.cc:2454] T eba78eb6d09d49ed9c364265e896e453 P 65582f059e8142f3ae495d608f61f6a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 690a0db9ce0d402985fd615d3d879679 in term 1.
I20260929 10:29:04.599690 31433 leader_election.cc:302] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 690a0db9ce0d402985fd615d3d879679, 76427dd6d6a64aa486fd5dafa71c087a; no voters:
I20260929 10:29:04.599872 31568 raft_consensus.cc:2781] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:29:04.600099 31568 raft_consensus.cc:686] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679 [term 1 LEADER]: Becoming Leader. State: Replica: 690a0db9ce0d402985fd615d3d879679, State: Running, Role: LEADER
I20260929 10:29:04.600461 31568 consensus_queue.cc:237] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "690a0db9ce0d402985fd615d3d879679" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 44107 } } peers { permanent_uuid: "65582f059e8142f3ae495d608f61f6a7" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 33083 } } peers { permanent_uuid: "76427dd6d6a64aa486fd5dafa71c087a" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 46877 } }
I20260929 10:29:04.603730 31078 catalog_manager.cc:5495] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679 reported cstate change: term changed from 0 to 1, leader changed from <none> to 690a0db9ce0d402985fd615d3d879679 (127.20.92.3). New cstate: current_term: 1 leader_uuid: "690a0db9ce0d402985fd615d3d879679" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "690a0db9ce0d402985fd615d3d879679" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 44107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65582f059e8142f3ae495d608f61f6a7" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 33083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76427dd6d6a64aa486fd5dafa71c087a" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 46877 } health_report { overall_health: UNKNOWN } } }
I20260929 10:29:04.618194 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:04.614421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48658 (local address 127.20.92.3:44107)
0929 10:29:04.614569 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:04.614574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:04.614589 (+ 15us) server_negotiation.cc:407] Connection header received
0929 10:29:04.614768 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:04.614770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:04.614827 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:04.614944 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:29:04.615187 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:04.615719 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:04.616592 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:04.616822 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:04.616934 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:04.617242 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:29:04.617695 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:29:04.617903 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 10:29:04.617951 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 10:29:04.618010 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 10:29:04.624874 31229 raft_consensus.cc:1261] T eba78eb6d09d49ed9c364265e896e453 P 65582f059e8142f3ae495d608f61f6a7 [term 1 FOLLOWER]: Refusing update from remote peer 690a0db9ce0d402985fd615d3d879679: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:29:04.624904 31363 raft_consensus.cc:1261] T eba78eb6d09d49ed9c364265e896e453 P 76427dd6d6a64aa486fd5dafa71c087a [term 1 FOLLOWER]: Refusing update from remote peer 690a0db9ce0d402985fd615d3d879679: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:29:04.625730 31570 consensus_queue.cc:1035] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65582f059e8142f3ae495d608f61f6a7" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 33083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:29:04.625990 31568 consensus_queue.cc:1035] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76427dd6d6a64aa486fd5dafa71c087a" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 46877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:29:04.634652 31574 mvcc.cc:205] Tried to move back new op lower bound from 7334616041975132160 to 7334616041886277632. Current Snapshot: MvccSnapshot[applied={T|T < 7334616041975132160}]
I20260929 10:29:04.642656 31083 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:29:04.642874 31083 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:29:04.645870 31083 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b34365c2e46f4fe1b65897715534a4ad: Sending DeleteTablet for 3 replicas of tablet eba78eb6d09d49ed9c364265e896e453
I20260929 10:29:04.646821 31343 tablet_service.cc:1514] Processing DeleteTablet for tablet eba78eb6d09d49ed9c364265e896e453 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:29:04 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:45412
I20260929 10:29:04.647110 31209 tablet_service.cc:1514] Processing DeleteTablet for tablet eba78eb6d09d49ed9c364265e896e453 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:29:04 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:39094
I20260929 10:29:04.647666 31584 tablet_replica.cc:330] T eba78eb6d09d49ed9c364265e896e453 P 65582f059e8142f3ae495d608f61f6a7: stopping tablet replica
I20260929 10:29:04.647704 31477 tablet_service.cc:1514] Processing DeleteTablet for tablet eba78eb6d09d49ed9c364265e896e453 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:29:04 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:48652
I20260929 10:29:04.648392 31585 tablet_replica.cc:330] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679: stopping tablet replica
I20260929 10:29:04.648702 31585 raft_consensus.cc:2229] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:29:04.649080 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 31146
I20260929 10:29:04.649099 31585 raft_consensus.cc:2258] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:29:04.649986 31583 tablet_replica.cc:330] T eba78eb6d09d49ed9c364265e896e453 P 76427dd6d6a64aa486fd5dafa71c087a: stopping tablet replica
I20260929 10:29:04.650336 31583 raft_consensus.cc:2229] T eba78eb6d09d49ed9c364265e896e453 P 76427dd6d6a64aa486fd5dafa71c087a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:29:04.650709 31583 raft_consensus.cc:2258] T eba78eb6d09d49ed9c364265e896e453 P 76427dd6d6a64aa486fd5dafa71c087a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:29:04.653136 31585 ts_tablet_manager.cc:1905] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:29:04.656566 31583 ts_tablet_manager.cc:1905] T eba78eb6d09d49ed9c364265e896e453 P 76427dd6d6a64aa486fd5dafa71c087a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:29:04.657342 31585 ts_tablet_manager.cc:1918] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:29:04.657680 31585 log.cc:1196] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677679287822-20848-0/minicluster-data/ts-2/wal/wals/eba78eb6d09d49ed9c364265e896e453
I20260929 10:29:04.658227 31585 ts_tablet_manager.cc:1939] T eba78eb6d09d49ed9c364265e896e453 P 690a0db9ce0d402985fd615d3d879679: Deleting consensus metadata
W20260929 10:29:04.658337 31070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.92.1:33083 (error 108)
I20260929 10:29:04.659237 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 31280
I20260929 10:29:04.660879 31071 catalog_manager.cc:4841] TS 690a0db9ce0d402985fd615d3d879679 (127.20.92.3:44107): tablet eba78eb6d09d49ed9c364265e896e453 (table test-table [id=cdd61829738d4b23a462065ae1a54396]) successfully deleted
I20260929 10:29:04.662583 31558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:04.662246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:33083 (local address 127.0.0.1:39098)
0929 10:29:04.662437 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 10:29:04.662505 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.1:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260929 10:29:04.662868 31070 catalog_manager.cc:4584] TS 65582f059e8142f3ae495d608f61f6a7 (127.20.92.1:33083): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eba78eb6d09d49ed9c364265e896e453: Network error: Client connection negotiation failed: client connection to 127.20.92.1:33083: connect: Connection refused (error 111)
I20260929 10:29:04.666925 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 31414
I20260929 10:29:04.669075 31558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:04.668780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:46877 (local address 127.0.0.1:45416)
0929 10:29:04.668944 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 10:29:04.668997 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.2:46877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 10:29:04.673077 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 31052
2026-09-29T10:29:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2532 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:29:04 dist-test-slave-9cj5 krb5kdc[31589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:29:04 dist-test-slave-9cj5 krb5kdc[31589](info): set up 2 sockets
Sep 29 10:29:04 dist-test-slave-9cj5 krb5kdc[31589](info): commencing operation
krb5kdc: starting...
W20260929 10:29:05.783584 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:29:05 dist-test-slave-9cj5 krb5kdc[31589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:29:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:29:05Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.20.92.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.20.92.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.20.92.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/rangeradmin_127.20.92.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.20.92.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/rangeradmin_127.20.92.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.20.92.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.20.92.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.20.92.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/rangerlookup_127.20.92.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.20.92.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/rangerlookup_127.20.92.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.92.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.92.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.92.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/HTTP_127.20.92.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.92.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/HTTP_127.20.92.20@KRBTEST.COM.keytab.
I20260929 10:29:05.977572 20848 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task8Q3Hzv/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/postgres -l logfile start
2026-09-29 10:29:09.075 UTC [31619] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 10:29:09.075 UTC [31619] LOG: listening on IPv4 address "127.20.92.20", port 56419
2026-09-29 10:29:09.080 UTC [31619] LOG: listening on Unix socket "/tmp/.s.PGSQL.56419"
2026-09-29 10:29:09.118 UTC [31622] LOG: database system was shut down at 2026-09-29 10:29:07 UTC
2026-09-29 10:29:09.123 UTC [31619] LOG: database system is ready to accept connections
I20260929 10:29:10.115531 20848 mini_postgres.cc:96] Postgres bound to 56419
2026-09-29 10:29:10.122 UTC [31630] FATAL: database "slave" does not exist
127.20.92.20:56419 - accepting connections
I20260929 10:29:10.123217 20848 mini_ranger.cc:161] Starting Ranger...
I20260929 10:29:10.140417 20848 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 10:29:10.809079 20848 mini_ranger.cc:87] Created ranger Postgres database
I20260929 10:29:10.809201 20848 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ranger-admin
2026-09-29 10:29:11,113 [I] DB FLAVOR :POSTGRES
2026-09-29 10:29:11,113 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 10:29:11,113 [I] Checking connection..
2026-09-29 10:29:11,113 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:56419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 10:29:11,361 [I] Checking connection passed.
2026-09-29 10:29:11,361 [I] --------- Verifying version history table ---------
2026-09-29 10:29:11,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:56419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 10:29:11,618 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 10:29:11,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:56419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 10:29:11,863 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 10:29:11,863 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 10:29:11,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:56419/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 10:29:12.126 UTC [31694] WARNING: there is no transaction in progress
2026-09-29 10:29:12,134 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 10:29:12,134 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:56419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 10:29:12,410 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 10:29:12,411 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 10:29:12,411 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:56419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 10:29:12,682 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:56419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 10:29:12,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:56419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9cj5.c.gcp-upstream.internal','N') ;"
2026-09-29 10:29:13,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:56419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 10:29:13,452 [I] Table x_portal_user does not exist in database ranger
2026-09-29 10:29:13,452 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:56419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 10:29:13,716 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 10:29:13,717 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:56419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 10:29:13,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:56419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 10:29:14,246 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 10:29:14,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:56419/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 10:29:15.336 UTC [31802] WARNING: there is no transaction in progress
2026-09-29 10:29:15.354 UTC [31802] WARNING: there is no transaction in progress
2026-09-29 10:29:15.373 UTC [31802] WARNING: there is no transaction in progress
2026-09-29 10:29:15.392 UTC [31802] WARNING: there is no transaction in progress
2026-09-29 10:29:15.410 UTC [31802] WARNING: there is no transaction in progress
2026-09-29 10:29:15.432 UTC [31802] WARNING: there is no transaction in progress
2026-09-29 10:29:15.504 UTC [31802] WARNING: there is no transaction in progress
2026-09-29 10:29:15.512 UTC [31802] WARNING: there is no transaction in progress
2026-09-29 10:29:15.528 UTC [31802] WARNING: there is no transaction in progress
2026-09-29 10:29:15.539 UTC [31802] WARNING: there is no transaction in progress
2026-09-29 10:29:15.551 UTC [31802] WARNING: there is no transaction in progress
2026-09-29 10:29:15.567 UTC [31802] WARNING: there is no transaction in progress
2026-09-29 10:29:15.578 UTC [31802] WARNING: there is no transaction in progress
2026-09-29 10:29:15.586 UTC [31802] WARNING: there is no transaction in progress
2026-09-29 10:29:15.594 UTC [31802] WARNING: there is no transaction in progress
2026-09-29 10:29:15.602 UTC [31802] WARNING: there is no transaction in progress
2026-09-29 10:29:16,343 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 10:29:16,343 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:56419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 10:29:16,588 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 10:29:16,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:56419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 10:29:16,859 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 10:29:16,859 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:56419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 10:29:17,114 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 10:29:17,114 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:56419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 10:29:17,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:56419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 10:29:17,669 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:56419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9cj5.c.gcp-upstream.internal';"
2026-09-29 10:29:17,928 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 10:29:17,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:56419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 10:29:18,199 [I] DB_PATCHES have already been applied
I20260929 10:29:18.208472 20848 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ranger-admin:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task8Q3Hzv/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 10:29:18.208566 20848 mini_ranger.cc:193] Using host: 127.20.92.20
I20260929 10:29:18.212978 20848 mini_ranger.cc:227] Ranger admin URL: http://127.20.92.20:34455
Sep 29, 2026 10:29:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 10:29:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 10:29:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 10:29:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 10:29:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 10:29:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 10:29:19 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 10:29:19.357299 20848 mini_ranger.cc:160] Time spent starting Ranger: real 9.234s user 0.000s sys 0.012s
Sep 29, 2026 10:29:41 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 10:29:41 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 29, 2026 10:29:41 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 10:29:41 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 10:29:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 10:29:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 10:29:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 10:29:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 10:29:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 10:29:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 10:29:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 10:29:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 10:29:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 10:29:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 10:29:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 10:29:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 10:29:41 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 10:29:41 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 10:29:42.592291 20848 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.20.92.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.20.92.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.20.92.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/rangerkms_127.20.92.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.20.92.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/rangerkms_127.20.92.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.92.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.92.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.92.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/HTTP_127.20.92.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.92.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/HTTP_127.20.92.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 10:29:42 dist-test-slave-9cj5 krb5kdc[31589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677782, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 10:29:42.690042 20848 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 10:29:42.701493 20848 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 10:29:43.990186 20848 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 10:29:43.990314 20848 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ranger-kms
I20260929 10:29:43.990432 20848 mini_ranger_kms.cc:227] Using postgres at 127.20.92.20:56419
2026-09-29 10:29:44,202 [I] DB FLAVOR :POSTGRES
2026-09-29 10:29:44,202 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 10:29:44,202 [I] Checking connection
2026-09-29 10:29:44,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:56419/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 10:29:44,506 [I] connection success
2026-09-29 10:29:44,506 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 10:29:44,507 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 10:29:44,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:56419/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 10:29:44,816 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 10:29:44,816 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 10:29:44,816 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 10:29:44,816 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:56419/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 10:29:45,209 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 10:29:45.706391 20848 mini_ranger_kms.cc:310] Created kms service
I20260929 10:29:45.902304 20848 mini_ranger_kms.cc:326] Created kudu user
I20260929 10:29:45.948732 20848 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 10:29:45 dist-test-slave-9cj5 krb5kdc[31589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677782, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.20.92.20@KRBTEST.COM
I20260929 10:29:46.377830 20848 mini_ranger_kms.cc:384] Added ranger policy
I20260929 10:29:46.377990 20848 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ranger-kms:/tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 10:29:46.378013 20848 mini_ranger_kms.cc:240] Using host: 127.20.92.20
I20260929 10:29:46.382069 20848 mini_ranger_kms.cc:276] Ranger KMS PID: 31983
I20260929 10:29:46.382299 20848 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.20.92.20:54049
Sep 29, 2026 10:29:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 10:29:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 10:29:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 10:29:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 10:29:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 10:29:48 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 10:29:48.822829 20848 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.133s user 0.004s sys 0.188s
I20260929 10:29:48.822973 20848 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 10:29:48.823091 20848 mini_ranger_kms.cc:401] 127.20.92.20:54049/kms/v1/keys
Sep 29 10:29:49 dist-test-slave-9cj5 krb5kdc[31589](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790677789, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.92.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 10:29:49 dist-test-slave-9cj5 krb5kdc[31589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677789, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.92.20@KRBTEST.COM for HTTP/127.20.92.20@KRBTEST.COM
Sep 29 10:29:49 dist-test-slave-9cj5 krb5kdc[31589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677789, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.92.20@KRBTEST.COM for HTTP/127.20.92.20@KRBTEST.COM
Sep 29 10:29:52 dist-test-slave-9cj5 krb5kdc[31589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.20.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
I20260929 10:29:52.229967 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:35923
--webserver_interface=127.20.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:35313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:35923
--ranger_config_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.92.20:54049/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:29:52.341929 32033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:29:52.342173 32033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:29:52.342212 32033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:29:52.346256 32033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:29:52.346306 32033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:29:52.346325 32033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:29:52.346341 32033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:29:52.346383 32033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:29:52.351627 32033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:35313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/master-0/wal
--ranger_kms_url=127.20.92.20:54049/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:35923
--ranger_config_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:35923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:29:52.352602 32033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:29:52.353690 32033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:29:52.362673 32038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:29:52.362739 32041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:29:52.362838 32033 server_base.cc:1034] running on GCE node
W20260929 10:29:52.362711 32039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:29:52.363420 32033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:29:52.364508 32033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:29:52.365667 32033 hybrid_clock.cc:648] HybridClock initialized: now 1790677792365640 us; error 39 us; skew 500 ppm
Sep 29 10:29:52 dist-test-slave-9cj5 krb5kdc[31589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677792, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:29:52.368511 32033 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:29:52.369372 32033 webserver.cc:460] Webserver started at http://127.20.92.62:34329/ using document root <none> and password file <none>
I20260929 10:29:52.369961 32033 fs_manager.cc:359] Metadata directory not provided
I20260929 10:29:52.370016 32033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:29:52.370225 32033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 10:29:52 dist-test-slave-9cj5 krb5kdc[31589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677792, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for HTTP/127.20.92.20@KRBTEST.COM
I20260929 10:29:52.437251 32033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "09d99dd4320544df97e466286f4fd557"
format_stamp: "Formatted at 2026-09-29 10:29:52 on dist-test-slave-9cj5"
server_key: "44704af2a4aaf8d3729eedab642d7a0d"
server_key_iv: "ccdabc19527c104b2c58e58e2d76f507"
server_key_version: "kuduclusterkey@0"
I20260929 10:29:52.437937 32033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "09d99dd4320544df97e466286f4fd557"
format_stamp: "Formatted at 2026-09-29 10:29:52 on dist-test-slave-9cj5"
server_key: "44704af2a4aaf8d3729eedab642d7a0d"
server_key_iv: "ccdabc19527c104b2c58e58e2d76f507"
server_key_version: "kuduclusterkey@0"
I20260929 10:29:52.441704 32033 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:29:52.460980 32048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:52.462493 32033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:29:52.462651 32033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "09d99dd4320544df97e466286f4fd557"
format_stamp: "Formatted at 2026-09-29 10:29:52 on dist-test-slave-9cj5"
server_key: "44704af2a4aaf8d3729eedab642d7a0d"
server_key_iv: "ccdabc19527c104b2c58e58e2d76f507"
server_key_version: "kuduclusterkey@0"
I20260929 10:29:52.462796 32033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:29:52.478785 32033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:29:52.482059 32033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:29:52.482255 32033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:29:52.491098 32033 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:35923
I20260929 10:29:52.491123 32100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:35923 every 8 connection(s)
I20260929 10:29:52.492115 32033 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:29:52.495582 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:29:52.498006 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 32033
I20260929 10:29:52.498149 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:29:52.498445 20848 external_mini_cluster.cc:1428] Setting key 6e5a60d88e80d2f958b4c7814e075027
I20260929 10:29:52.503048 32101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09d99dd4320544df97e466286f4fd557: Bootstrap starting.
I20260929 10:29:52.506074 32101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09d99dd4320544df97e466286f4fd557: Neither blocks nor log segments found. Creating new log.
I20260929 10:29:52.507047 32101 log.cc:824] T 00000000000000000000000000000000 P 09d99dd4320544df97e466286f4fd557: Log is configured to *not* fsync() on all Append() calls
Sep 29 10:29:52 dist-test-slave-9cj5 krb5kdc[31589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:29:52.509479 32101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09d99dd4320544df97e466286f4fd557: No bootstrap required, opened a new log
I20260929 10:29:52.513221 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:52.499905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56192 (local address 127.20.92.62:35923)
0929 10:29:52.500319 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:52.500335 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:52.500371 (+ 36us) server_negotiation.cc:407] Connection header received
0929 10:29:52.501021 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:52.501037 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:52.501373 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:52.501713 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:29:52.502568 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:52.503608 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:52.504462 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:52.505075 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:52.505269 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:52.508009 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:29:52.508015 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:29:52.508030 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:29:52.508058 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:29:52.510349 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:29:52.510831 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:29:52.510836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:29:52.510841 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:29:52.510944 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:29:52.511247 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:29:52.511252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:29:52.511254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:29:52.511666 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:29:52.511947 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0929 10:29:52.512136 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 10:29:52.512456 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":132,"threads_started":1}
I20260929 10:29:52.513685 32101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09d99dd4320544df97e466286f4fd557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09d99dd4320544df97e466286f4fd557" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 35923 } }
I20260929 10:29:52.514006 32101 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09d99dd4320544df97e466286f4fd557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:29:52.514058 32101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09d99dd4320544df97e466286f4fd557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09d99dd4320544df97e466286f4fd557, State: Initialized, Role: FOLLOWER
I20260929 10:29:52.514537 32101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09d99dd4320544df97e466286f4fd557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09d99dd4320544df97e466286f4fd557" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 35923 } }
I20260929 10:29:52.514696 32101 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09d99dd4320544df97e466286f4fd557 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:29:52.514768 32101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09d99dd4320544df97e466286f4fd557 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:29:52.514859 32101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09d99dd4320544df97e466286f4fd557 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:29:52.516037 32101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09d99dd4320544df97e466286f4fd557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09d99dd4320544df97e466286f4fd557" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 35923 } }
I20260929 10:29:52.516424 32101 leader_election.cc:302] T 00000000000000000000000000000000 P 09d99dd4320544df97e466286f4fd557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09d99dd4320544df97e466286f4fd557; no voters:
I20260929 10:29:52.516816 32101 leader_election.cc:288] T 00000000000000000000000000000000 P 09d99dd4320544df97e466286f4fd557 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:29:52.517011 32106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09d99dd4320544df97e466286f4fd557 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:29:52.517333 32106 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09d99dd4320544df97e466286f4fd557 [term 1 LEADER]: Becoming Leader. State: Replica: 09d99dd4320544df97e466286f4fd557, State: Running, Role: LEADER
I20260929 10:29:52.517804 32106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09d99dd4320544df97e466286f4fd557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09d99dd4320544df97e466286f4fd557" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 35923 } }
I20260929 10:29:52.518203 32101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09d99dd4320544df97e466286f4fd557 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:29:52.519727 32108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d99dd4320544df97e466286f4fd557 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09d99dd4320544df97e466286f4fd557. Latest consensus state: current_term: 1 leader_uuid: "09d99dd4320544df97e466286f4fd557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09d99dd4320544df97e466286f4fd557" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 35923 } } }
I20260929 10:29:52.519873 32108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d99dd4320544df97e466286f4fd557 [sys.catalog]: This master's current role is: LEADER
I20260929 10:29:52.520267 32107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d99dd4320544df97e466286f4fd557 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09d99dd4320544df97e466286f4fd557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09d99dd4320544df97e466286f4fd557" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 35923 } } }
I20260929 10:29:52.520367 32107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d99dd4320544df97e466286f4fd557 [sys.catalog]: This master's current role is: LEADER
I20260929 10:29:52.520679 32101 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 10:29:52.522521 32117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:29:52.528970 32117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:29:52.539777 32117 catalog_manager.cc:1347] Generated new cluster ID: d0a1f0b41a6942efb57188d248130fa0
I20260929 10:29:52.539850 32117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:29:52.560956 32117 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:29:52.562835 32117 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:29:52.567698 32117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09d99dd4320544df97e466286f4fd557: Generated new TSK 0
I20260929 10:29:52.568578 32117 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
I20260929 10:29:53.293473 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.92.20:54049/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:0
--local_ip_for_outbound_sockets=127.20.92.1
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:35923
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:35313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:29:53.458902 32146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:29:53.459267 32146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:29:53.459334 32146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:29:53.465478 32146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:29:53.465569 32146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:29:53.465664 32146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:29:53.474342 32146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:35313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.20.92.20:54049/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:35923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:29:53.475759 32146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:29:53.477089 32146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:29:53.489970 32154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:29:53.490281 32146 server_base.cc:1034] running on GCE node
W20260929 10:29:53.490329 32153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:29:53.490477 32156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:29:53.493338 32146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:29:53.494182 32146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:29:53.495364 32146 hybrid_clock.cc:648] HybridClock initialized: now 1790677793495328 us; error 56 us; skew 500 ppm
Sep 29 10:29:53 dist-test-slave-9cj5 krb5kdc[31589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677793, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:29:53.503248 32146 init.cc:377] Logged in from keytab as kudu/127.20.92.1@KRBTEST.COM (short username kudu)
I20260929 10:29:53.504251 32146 webserver.cc:460] Webserver started at http://127.20.92.1:44871/ using document root <none> and password file <none>
I20260929 10:29:53.504840 32146 fs_manager.cc:359] Metadata directory not provided
I20260929 10:29:53.504901 32146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:29:53.505125 32146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 10:29:53 dist-test-slave-9cj5 krb5kdc[31589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677793, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for HTTP/127.20.92.20@KRBTEST.COM
I20260929 10:29:53.523082 32146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-0/data/instance:
uuid: "021b306ba910400ab06428ccb5c55861"
format_stamp: "Formatted at 2026-09-29 10:29:53 on dist-test-slave-9cj5"
server_key: "97cac2328f8e79a83741e76697c10836"
server_key_iv: "b66d61a36e079582647323683476372b"
server_key_version: "kuduclusterkey@0"
I20260929 10:29:53.524008 32146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance:
uuid: "021b306ba910400ab06428ccb5c55861"
format_stamp: "Formatted at 2026-09-29 10:29:53 on dist-test-slave-9cj5"
server_key: "97cac2328f8e79a83741e76697c10836"
server_key_iv: "b66d61a36e079582647323683476372b"
server_key_version: "kuduclusterkey@0"
I20260929 10:29:53.529155 32146 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 10:29:53.558112 32163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:53.559937 32146 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 10:29:53.560079 32146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "021b306ba910400ab06428ccb5c55861"
format_stamp: "Formatted at 2026-09-29 10:29:53 on dist-test-slave-9cj5"
server_key: "97cac2328f8e79a83741e76697c10836"
server_key_iv: "b66d61a36e079582647323683476372b"
server_key_version: "kuduclusterkey@0"
I20260929 10:29:53.560228 32146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:29:53.575142 32146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:29:53.578810 32146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:29:53.579020 32146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:29:53.579674 32146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:29:53.580693 32146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:29:53.580778 32146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:53.580847 32146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:29:53.580879 32146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:53.591749 32146 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:35833
I20260929 10:29:53.592847 32146 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
I20260929 10:29:53.595082 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 32146
I20260929 10:29:53.595237 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance
I20260929 10:29:53.595587 20848 external_mini_cluster.cc:1428] Setting key bde0e818a5a453821d6bcd4cbdeb221c
WARNING: no policy specified for kudu/127.20.92.2@KRBTEST.COM; defaulting to no policy
I20260929 10:29:53.610405 32278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:35833 every 8 connection(s)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.2@KRBTEST.COM" created.
Sep 29 10:29:53 dist-test-slave-9cj5 krb5kdc[31589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677793, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:29:53.630669 32284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:53.613272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:58425 (local address 127.20.92.62:35923)
0929 10:29:53.614888 (+ 1616us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:53.614893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:53.614913 (+ 20us) server_negotiation.cc:407] Connection header received
0929 10:29:53.615347 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:53.615351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:53.615416 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:53.615510 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:29:53.617409 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:53.618421 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:53.620400 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:53.620811 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:53.620941 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:53.626173 (+ 5232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:29:53.626177 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:29:53.626179 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:29:53.626204 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:29:53.628128 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:29:53.628779 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:29:53.628784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:29:53.628786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:29:53.628870 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:29:53.629331 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:29:53.629336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:29:53.629338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:29:53.629585 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:29:53.629697 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 10:29:53.630325 (+ 628us) server_negotiation.cc:299] Negotiation successful
0929 10:29:53.630477 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1534,"thread_start_us":90,"threads_started":1}
I20260929 10:29:53.631587 32282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:53.612346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:35923 (local address 127.20.92.1:58425)
0929 10:29:53.613642 (+ 1296us) negotiation.cc:107] Waiting for socket to connect
0929 10:29:53.613707 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 10:29:53.614906 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:29:53.615873 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:29:53.615884 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:29:53.616412 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:29:53.617180 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:53.617207 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:53.618645 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:53.618652 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:53.620222 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:53.620239 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:53.621171 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:53.621175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:53.621403 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:53.622223 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:29:53.622248 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:29:53.625948 (+ 3700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:29:53.628258 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:29:53.628265 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:29:53.628284 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:29:53.628664 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:29:53.629057 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:29:53.629060 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:29:53.629062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:29:53.629230 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:29:53.629695 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:29:53.629702 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:29:53.630072 (+ 370us) client_negotiation.cc:769] Sending connection context
0929 10:29:53.630273 (+ 201us) client_negotiation.cc:240] Negotiation successful
0929 10:29:53.630604 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1037,"thread_start_us":166,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
I20260929 10:29:53.632995 32279 heartbeater.cc:347] Connected to a master server at 127.20.92.62:35923
I20260929 10:29:53.633255 32279 heartbeater.cc:464] Registering TS with master...
I20260929 10:29:53.633553 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.92.20:54049/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:0
--local_ip_for_outbound_sockets=127.20.92.2
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:35923
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:35313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:29:53.633884 32279 heartbeater.cc:511] Master 127.20.92.62:35923 requested a full tablet report, sending...
I20260929 10:29:53.635568 32065 ts_manager.cc:194] Registered new tserver with Master: 021b306ba910400ab06428ccb5c55861 (127.20.92.1:35833)
I20260929 10:29:53.636663 32065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.1@KRBTEST.COM'} at 127.20.92.1:58425
Sep 29 10:29:53 dist-test-slave-9cj5 krb5kdc[31589](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790677793, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 10:29:53.764585 32285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:29:53.764990 32285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:29:53.765070 32285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:29:53.770236 32285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:29:53.770323 32285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:29:53.770443 32285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:29:53.776028 32285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:35313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.20.92.20:54049/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:35923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:29:53.777220 32285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:29:53.778329 32285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:29:53.788432 32294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:29:53.788477 32292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:29:53.788920 32291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:29:53.789268 32285 server_base.cc:1034] running on GCE node
I20260929 10:29:53.789796 32285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:29:53.790663 32285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:29:53.791857 32285 hybrid_clock.cc:648] HybridClock initialized: now 1790677793791827 us; error 52 us; skew 500 ppm
Sep 29 10:29:53 dist-test-slave-9cj5 krb5kdc[31589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677793, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:29:53.795164 32285 init.cc:377] Logged in from keytab as kudu/127.20.92.2@KRBTEST.COM (short username kudu)
I20260929 10:29:53.796195 32285 webserver.cc:460] Webserver started at http://127.20.92.2:43829/ using document root <none> and password file <none>
I20260929 10:29:53.796778 32285 fs_manager.cc:359] Metadata directory not provided
I20260929 10:29:53.796871 32285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:29:53.797135 32285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 10:29:53 dist-test-slave-9cj5 krb5kdc[31589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677793, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for HTTP/127.20.92.20@KRBTEST.COM
I20260929 10:29:53.820634 32285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-1/data/instance:
uuid: "eaad6afddabf465b95f89f9fff498e7a"
format_stamp: "Formatted at 2026-09-29 10:29:53 on dist-test-slave-9cj5"
server_key: "300e3478151c1d8b78e734442d3600f9"
server_key_iv: "042d15d7bdccc32e8e75e63a16ad1f9e"
server_key_version: "kuduclusterkey@0"
I20260929 10:29:53.821491 32285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance:
uuid: "eaad6afddabf465b95f89f9fff498e7a"
format_stamp: "Formatted at 2026-09-29 10:29:53 on dist-test-slave-9cj5"
server_key: "300e3478151c1d8b78e734442d3600f9"
server_key_iv: "042d15d7bdccc32e8e75e63a16ad1f9e"
server_key_version: "kuduclusterkey@0"
I20260929 10:29:53.827725 32285 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.003s
I20260929 10:29:53.854322 32301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:53.856364 32285 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.002s sys 0.000s
I20260929 10:29:53.856546 32285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "eaad6afddabf465b95f89f9fff498e7a"
format_stamp: "Formatted at 2026-09-29 10:29:53 on dist-test-slave-9cj5"
server_key: "300e3478151c1d8b78e734442d3600f9"
server_key_iv: "042d15d7bdccc32e8e75e63a16ad1f9e"
server_key_version: "kuduclusterkey@0"
I20260929 10:29:53.856714 32285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:29:53.878829 32285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:29:53.882591 32285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:29:53.882798 32285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:29:53.883528 32285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:29:53.884548 32285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:29:53.884615 32285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:53.884692 32285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:29:53.884724 32285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:53.899945 32285 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:39695
I20260929 10:29:53.901140 32285 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
I20260929 10:29:53.902532 32414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:39695 every 8 connection(s)
I20260929 10:29:53.905659 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 32285
I20260929 10:29:53.905804 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance
I20260929 10:29:53.906136 20848 external_mini_cluster.cc:1428] Setting key 1a241e523f3637a152cd1e6e071c2ad3
Sep 29 10:29:53 dist-test-slave-9cj5 krb5kdc[31589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677793, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.92.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.3@KRBTEST.COM" created.
I20260929 10:29:53.925885 32284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:53.906079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:49951 (local address 127.20.92.62:35923)
0929 10:29:53.906218 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:53.906225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:53.909691 (+ 3466us) server_negotiation.cc:407] Connection header received
0929 10:29:53.910955 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:53.910958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:53.911011 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:53.911096 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:29:53.912713 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:53.913518 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:53.915339 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:53.915712 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:53.915806 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:53.918911 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:29:53.918915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:29:53.918917 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:29:53.918948 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:29:53.920778 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:29:53.921550 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:29:53.921554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:29:53.921556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:29:53.921655 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:29:53.922229 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:29:53.922232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:29:53.922234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:29:53.922464 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:29:53.922549 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 10:29:53.925496 (+ 2947us) server_negotiation.cc:299] Negotiation successful
0929 10:29:53.925687 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 10:29:53.927621 32417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:53.904944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:35923 (local address 127.20.92.2:49951)
0929 10:29:53.909498 (+ 4554us) negotiation.cc:107] Waiting for socket to connect
0929 10:29:53.909570 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 10:29:53.910702 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:29:53.911448 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:29:53.911459 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:29:53.911835 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:29:53.912527 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:53.912547 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:53.913694 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:53.913697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:53.915199 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:53.915209 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:53.916031 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:53.916035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:53.916249 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:53.916959 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:29:53.916980 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:29:53.918703 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:29:53.921041 (+ 2338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:29:53.921048 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:29:53.921070 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:29:53.921428 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:29:53.921796 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:29:53.921799 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:29:53.921801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:29:53.922111 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:29:53.924759 (+ 2648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:29:53.924768 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:29:53.925178 (+ 410us) client_negotiation.cc:769] Sending connection context
0929 10:29:53.926139 (+ 961us) client_negotiation.cc:240] Negotiation successful
0929 10:29:53.926517 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4293,"thread_start_us":133,"threads_started":1}
I20260929 10:29:53.928963 32415 heartbeater.cc:347] Connected to a master server at 127.20.92.62:35923
I20260929 10:29:53.929216 32415 heartbeater.cc:464] Registering TS with master...
I20260929 10:29:53.929849 32415 heartbeater.cc:511] Master 127.20.92.62:35923 requested a full tablet report, sending...
I20260929 10:29:53.931013 32065 ts_manager.cc:194] Registered new tserver with Master: eaad6afddabf465b95f89f9fff498e7a (127.20.92.2:39695)
I20260929 10:29:53.931795 32065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.2@KRBTEST.COM'} at 127.20.92.2:49951
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
I20260929 10:29:53.954641 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.92.20:54049/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:0
--local_ip_for_outbound_sockets=127.20.92.3
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:35923
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:35313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:29:54.115985 32420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:29:54.116379 32420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:29:54.116451 32420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:29:54.123047 32420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:29:54.123145 32420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:29:54.123250 32420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:29:54.131354 32420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:35313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.20.92.20:54049/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:35923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:29:54.132818 32420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:29:54.134202 32420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:29:54.147874 32426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:29:54.148316 32420 server_base.cc:1034] running on GCE node
W20260929 10:29:54.148623 32425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:29:54.150282 32428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:29:54.150781 32420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:29:54.151605 32420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:29:54.152808 32420 hybrid_clock.cc:648] HybridClock initialized: now 1790677794152743 us; error 79 us; skew 500 ppm
Sep 29 10:29:54 dist-test-slave-9cj5 krb5kdc[31589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677794, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:29:54.156445 32420 init.cc:377] Logged in from keytab as kudu/127.20.92.3@KRBTEST.COM (short username kudu)
I20260929 10:29:54.157527 32420 webserver.cc:460] Webserver started at http://127.20.92.3:36717/ using document root <none> and password file <none>
I20260929 10:29:54.158268 32420 fs_manager.cc:359] Metadata directory not provided
I20260929 10:29:54.158519 32420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:29:54.159109 32420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 10:29:54 dist-test-slave-9cj5 krb5kdc[31589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677794, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for HTTP/127.20.92.20@KRBTEST.COM
I20260929 10:29:54.177948 32420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-2/data/instance:
uuid: "c241085ff3bf4461b00a253bb9627f6b"
format_stamp: "Formatted at 2026-09-29 10:29:54 on dist-test-slave-9cj5"
server_key: "79aa26a677598ea11a7704c61f461bf7"
server_key_iv: "dc62ca556ea28250fb25282173c3d4a0"
server_key_version: "kuduclusterkey@0"
I20260929 10:29:54.178707 32420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance:
uuid: "c241085ff3bf4461b00a253bb9627f6b"
format_stamp: "Formatted at 2026-09-29 10:29:54 on dist-test-slave-9cj5"
server_key: "79aa26a677598ea11a7704c61f461bf7"
server_key_iv: "dc62ca556ea28250fb25282173c3d4a0"
server_key_version: "kuduclusterkey@0"
I20260929 10:29:54.186360 32420 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260929 10:29:54.206179 32435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:54.208032 32420 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 10:29:54.208170 32420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "c241085ff3bf4461b00a253bb9627f6b"
format_stamp: "Formatted at 2026-09-29 10:29:54 on dist-test-slave-9cj5"
server_key: "79aa26a677598ea11a7704c61f461bf7"
server_key_iv: "dc62ca556ea28250fb25282173c3d4a0"
server_key_version: "kuduclusterkey@0"
I20260929 10:29:54.208304 32420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 29 10:29:54 dist-test-slave-9cj5 krb5kdc[31589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677793, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for HTTP/127.20.92.20@KRBTEST.COM
I20260929 10:29:54.292279 32420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:29:54.296088 32420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:29:54.296331 32420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:29:54.296989 32420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:29:54.298072 32420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:29:54.298172 32420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:54.298264 32420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:29:54.298324 32420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:29:54.311452 32420 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:34817
I20260929 10:29:54.312649 32420 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:29:54.314301 32548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:34817 every 8 connection(s)
I20260929 10:29:54.319963 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 32420
I20260929 10:29:54.320078 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance
I20260929 10:29:54.320365 20848 external_mini_cluster.cc:1428] Setting key 53800c8c5d73a48b305d2eec356c31dd
Sep 29 10:29:54 dist-test-slave-9cj5 krb5kdc[31589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677794, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:29:54.335623 32284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:54.317706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:42603 (local address 127.20.92.62:35923)
0929 10:29:54.317889 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:54.317894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:54.318890 (+ 996us) server_negotiation.cc:407] Connection header received
0929 10:29:54.320403 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:54.320407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:54.320487 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:54.320543 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:29:54.323959 (+ 3416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:54.324728 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:54.326405 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:54.326752 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:54.326833 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:54.329913 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:29:54.329916 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:29:54.329919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:29:54.329947 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:29:54.331801 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:29:54.332442 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:29:54.332446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:29:54.332448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:29:54.332534 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:29:54.334300 (+ 1766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:29:54.334303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:29:54.334305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:29:54.334526 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:29:54.334624 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 10:29:54.335315 (+ 691us) server_negotiation.cc:299] Negotiation successful
0929 10:29:54.335447 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 10:29:54.338524 32551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:54.316636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:35923 (local address 127.20.92.3:42603)
0929 10:29:54.318713 (+ 2077us) negotiation.cc:107] Waiting for socket to connect
0929 10:29:54.318767 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 10:29:54.320015 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:29:54.322733 (+ 2718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:29:54.322743 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:29:54.323099 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:29:54.323788 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:54.323809 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:54.324884 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:54.324887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:54.326264 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:54.326276 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:54.327035 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:54.327039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:54.327231 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:54.327958 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:29:54.327978 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:29:54.329674 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:29:54.331945 (+ 2271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:29:54.331951 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:29:54.331967 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:29:54.332318 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:29:54.332672 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:29:54.332675 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:29:54.332677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:29:54.333466 (+ 789us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:29:54.334694 (+ 1228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:29:54.334701 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:29:54.335033 (+ 332us) client_negotiation.cc:769] Sending connection context
0929 10:29:54.336177 (+ 1144us) client_negotiation.cc:240] Negotiation successful
0929 10:29:54.336532 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1851,"thread_start_us":140,"threads_started":1}
I20260929 10:29:54.340052 32549 heartbeater.cc:347] Connected to a master server at 127.20.92.62:35923
I20260929 10:29:54.340312 32549 heartbeater.cc:464] Registering TS with master...
I20260929 10:29:54.340881 32549 heartbeater.cc:511] Master 127.20.92.62:35923 requested a full tablet report, sending...
I20260929 10:29:54.341958 32065 ts_manager.cc:194] Registered new tserver with Master: c241085ff3bf4461b00a253bb9627f6b (127.20.92.3:34817)
I20260929 10:29:54.342653 32065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.3@KRBTEST.COM'} at 127.20.92.3:42603
I20260929 10:29:54.346077 20848 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:29:54.359164 32284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:54.348917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56208 (local address 127.20.92.62:35923)
0929 10:29:54.349028 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:54.349032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:54.350043 (+ 1011us) server_negotiation.cc:407] Connection header received
0929 10:29:54.350157 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:54.350160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:54.350208 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:54.350297 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:29:54.351134 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:54.351974 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:54.353123 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:54.353469 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:54.353575 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:54.355260 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:29:54.355264 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:29:54.355267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:29:54.355293 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:29:54.357182 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:29:54.357708 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:29:54.357711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:29:54.357713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:29:54.357824 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:29:54.358144 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:29:54.358147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:29:54.358149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:29:54.358397 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:29:54.358503 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 10:29:54.358880 (+ 377us) server_negotiation.cc:299] Negotiation successful
0929 10:29:54.359002 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 10:29:54.363027 32065 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56208:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:29:54.367846 32065 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:29:54.387849 32564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:54.379181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:39695 (local address 127.0.0.1:33502)
0929 10:29:54.381588 (+ 2407us) negotiation.cc:107] Waiting for socket to connect
0929 10:29:54.381643 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 10:29:54.381901 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:29:54.382661 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:29:54.382667 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:29:54.382686 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:29:54.382833 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:54.382840 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:54.384330 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:54.384333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:54.385723 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:54.385735 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:54.387237 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:54.387241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:54.387392 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:54.387440 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 10:29:54.387559 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 10:29:54.387681 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2222,"thread_start_us":112,"threads_started":1}
I20260929 10:29:54.388458 32562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:54.379877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33502 (local address 127.20.92.2:39695)
0929 10:29:54.380203 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:54.380220 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:54.381733 (+ 1513us) server_negotiation.cc:407] Connection header received
0929 10:29:54.382099 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:54.382105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:54.382345 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:54.382480 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:29:54.382959 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:54.384190 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:54.385919 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:54.386889 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:54.387010 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:54.387169 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 10:29:54.388197 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0929 10:29:54.388316 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":76,"threads_started":1}
I20260929 10:29:54.389784 32561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:54.380642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:34817 (local address 127.0.0.1:49098)
0929 10:29:54.382411 (+ 1769us) negotiation.cc:107] Waiting for socket to connect
0929 10:29:54.382424 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:29:54.382502 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:29:54.383626 (+ 1124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:29:54.383629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:29:54.383645 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:29:54.383747 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:54.383767 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:54.386104 (+ 2337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:54.386107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:54.387477 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:54.387488 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:54.389379 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:54.389383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:54.389489 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:54.389510 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 10:29:54.389569 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 10:29:54.389631 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1706,"thread_start_us":129,"threads_started":1}
I20260929 10:29:54.390215 32565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:54.381113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49098 (local address 127.20.92.3:34817)
0929 10:29:54.383098 (+ 1985us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:54.383109 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:54.383130 (+ 21us) server_negotiation.cc:407] Connection header received
0929 10:29:54.383196 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:54.383201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:54.383372 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:54.383501 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:29:54.383883 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:54.385998 (+ 2115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:54.387632 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:54.388407 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:54.388502 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:54.389929 (+ 1427us) server_negotiation.cc:1035] Waiting for connection context
0929 10:29:54.389984 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 10:29:54.390083 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1882,"thread_start_us":95,"threads_started":1}
I20260929 10:29:54.391700 32346 tablet_service.cc:1467] Processing CreateTablet for tablet 1719b3f186b44dc5b442cd1264cec9e8 (DEFAULT_TABLE table=test-table [id=09d6f0824d244822ad4ce2b8863afcf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:29:54.395771 32563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:54.380895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:35833 (local address 127.0.0.1:37870)
0929 10:29:54.385641 (+ 4746us) negotiation.cc:107] Waiting for socket to connect
0929 10:29:54.385656 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:29:54.385732 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:29:54.391104 (+ 5372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:29:54.391107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:29:54.391124 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:29:54.391202 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:54.391208 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:54.392510 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:54.392513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:54.394387 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:54.394396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:54.395427 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:54.395431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:54.395513 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:54.395538 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 10:29:54.395580 (+ 42us) client_negotiation.cc:240] Negotiation successful
0929 10:29:54.395640 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4688,"thread_start_us":73,"threads_started":1}
I20260929 10:29:54.395717 32464 tablet_service.cc:1467] Processing CreateTablet for tablet 1719b3f186b44dc5b442cd1264cec9e8 (DEFAULT_TABLE table=test-table [id=09d6f0824d244822ad4ce2b8863afcf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:29:54.396440 32566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:54.381250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37870 (local address 127.20.92.1:35833)
0929 10:29:54.390519 (+ 9269us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:54.390526 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:54.390549 (+ 23us) server_negotiation.cc:407] Connection header received
0929 10:29:54.390625 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:54.390631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:54.390831 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:54.390966 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:29:54.391327 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:54.392389 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:54.394554 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:54.395300 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:54.396039 (+ 739us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:54.396177 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 10:29:54.396226 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 10:29:54.396311 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9160,"thread_start_us":100,"threads_started":1}
I20260929 10:29:54.396809 32464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1719b3f186b44dc5b442cd1264cec9e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:29:54.398011 32346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1719b3f186b44dc5b442cd1264cec9e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:29:54.399648 32203 tablet_service.cc:1467] Processing CreateTablet for tablet 1719b3f186b44dc5b442cd1264cec9e8 (DEFAULT_TABLE table=test-table [id=09d6f0824d244822ad4ce2b8863afcf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:29:54.400678 32203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1719b3f186b44dc5b442cd1264cec9e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:29:54.410981 32567 tablet_bootstrap.cc:492] T 1719b3f186b44dc5b442cd1264cec9e8 P 021b306ba910400ab06428ccb5c55861: Bootstrap starting.
I20260929 10:29:54.413359 32568 tablet_bootstrap.cc:492] T 1719b3f186b44dc5b442cd1264cec9e8 P eaad6afddabf465b95f89f9fff498e7a: Bootstrap starting.
I20260929 10:29:54.413415 32567 tablet_bootstrap.cc:654] T 1719b3f186b44dc5b442cd1264cec9e8 P 021b306ba910400ab06428ccb5c55861: Neither blocks nor log segments found. Creating new log.
I20260929 10:29:54.414474 32567 log.cc:824] T 1719b3f186b44dc5b442cd1264cec9e8 P 021b306ba910400ab06428ccb5c55861: Log is configured to *not* fsync() on all Append() calls
I20260929 10:29:54.415248 32569 tablet_bootstrap.cc:492] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b: Bootstrap starting.
I20260929 10:29:54.415796 32568 tablet_bootstrap.cc:654] T 1719b3f186b44dc5b442cd1264cec9e8 P eaad6afddabf465b95f89f9fff498e7a: Neither blocks nor log segments found. Creating new log.
I20260929 10:29:54.417127 32568 log.cc:824] T 1719b3f186b44dc5b442cd1264cec9e8 P eaad6afddabf465b95f89f9fff498e7a: Log is configured to *not* fsync() on all Append() calls
I20260929 10:29:54.418470 32569 tablet_bootstrap.cc:654] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b: Neither blocks nor log segments found. Creating new log.
I20260929 10:29:54.419296 32569 log.cc:824] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b: Log is configured to *not* fsync() on all Append() calls
I20260929 10:29:54.420178 32568 tablet_bootstrap.cc:492] T 1719b3f186b44dc5b442cd1264cec9e8 P eaad6afddabf465b95f89f9fff498e7a: No bootstrap required, opened a new log
I20260929 10:29:54.420397 32568 ts_tablet_manager.cc:1397] T 1719b3f186b44dc5b442cd1264cec9e8 P eaad6afddabf465b95f89f9fff498e7a: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260929 10:29:54.421286 32567 tablet_bootstrap.cc:492] T 1719b3f186b44dc5b442cd1264cec9e8 P 021b306ba910400ab06428ccb5c55861: No bootstrap required, opened a new log
I20260929 10:29:54.421478 32567 ts_tablet_manager.cc:1397] T 1719b3f186b44dc5b442cd1264cec9e8 P 021b306ba910400ab06428ccb5c55861: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
I20260929 10:29:54.424645 32568 raft_consensus.cc:348] T 1719b3f186b44dc5b442cd1264cec9e8 P eaad6afddabf465b95f89f9fff498e7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaad6afddabf465b95f89f9fff498e7a" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39695 } } peers { permanent_uuid: "c241085ff3bf4461b00a253bb9627f6b" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 34817 } } peers { permanent_uuid: "021b306ba910400ab06428ccb5c55861" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35833 } }
I20260929 10:29:54.424942 32568 raft_consensus.cc:374] T 1719b3f186b44dc5b442cd1264cec9e8 P eaad6afddabf465b95f89f9fff498e7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:29:54.425004 32568 raft_consensus.cc:729] T 1719b3f186b44dc5b442cd1264cec9e8 P eaad6afddabf465b95f89f9fff498e7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaad6afddabf465b95f89f9fff498e7a, State: Initialized, Role: FOLLOWER
I20260929 10:29:54.425534 32568 consensus_queue.cc:260] T 1719b3f186b44dc5b442cd1264cec9e8 P eaad6afddabf465b95f89f9fff498e7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaad6afddabf465b95f89f9fff498e7a" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39695 } } peers { permanent_uuid: "c241085ff3bf4461b00a253bb9627f6b" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 34817 } } peers { permanent_uuid: "021b306ba910400ab06428ccb5c55861" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35833 } }
I20260929 10:29:54.425806 32567 raft_consensus.cc:348] T 1719b3f186b44dc5b442cd1264cec9e8 P 021b306ba910400ab06428ccb5c55861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaad6afddabf465b95f89f9fff498e7a" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39695 } } peers { permanent_uuid: "c241085ff3bf4461b00a253bb9627f6b" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 34817 } } peers { permanent_uuid: "021b306ba910400ab06428ccb5c55861" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35833 } }
I20260929 10:29:54.426060 32567 raft_consensus.cc:374] T 1719b3f186b44dc5b442cd1264cec9e8 P 021b306ba910400ab06428ccb5c55861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:29:54.426113 32567 raft_consensus.cc:729] T 1719b3f186b44dc5b442cd1264cec9e8 P 021b306ba910400ab06428ccb5c55861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021b306ba910400ab06428ccb5c55861, State: Initialized, Role: FOLLOWER
I20260929 10:29:54.426529 32567 consensus_queue.cc:260] T 1719b3f186b44dc5b442cd1264cec9e8 P 021b306ba910400ab06428ccb5c55861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaad6afddabf465b95f89f9fff498e7a" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39695 } } peers { permanent_uuid: "c241085ff3bf4461b00a253bb9627f6b" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 34817 } } peers { permanent_uuid: "021b306ba910400ab06428ccb5c55861" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35833 } }
I20260929 10:29:54.426692 32568 ts_tablet_manager.cc:1428] T 1719b3f186b44dc5b442cd1264cec9e8 P eaad6afddabf465b95f89f9fff498e7a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 10:29:54.427052 32415 heartbeater.cc:503] Master 127.20.92.62:35923 was elected leader, sending a full tablet report...
I20260929 10:29:54.427487 32567 ts_tablet_manager.cc:1428] T 1719b3f186b44dc5b442cd1264cec9e8 P 021b306ba910400ab06428ccb5c55861: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260929 10:29:54.427939 32279 heartbeater.cc:503] Master 127.20.92.62:35923 was elected leader, sending a full tablet report...
I20260929 10:29:54.431130 32569 tablet_bootstrap.cc:492] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b: No bootstrap required, opened a new log
I20260929 10:29:54.431360 32569 ts_tablet_manager.cc:1397] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20260929 10:29:54.435442 32569 raft_consensus.cc:348] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaad6afddabf465b95f89f9fff498e7a" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39695 } } peers { permanent_uuid: "c241085ff3bf4461b00a253bb9627f6b" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 34817 } } peers { permanent_uuid: "021b306ba910400ab06428ccb5c55861" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35833 } }
I20260929 10:29:54.435675 32569 raft_consensus.cc:374] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:29:54.435714 32569 raft_consensus.cc:729] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c241085ff3bf4461b00a253bb9627f6b, State: Initialized, Role: FOLLOWER
I20260929 10:29:54.436143 32569 consensus_queue.cc:260] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaad6afddabf465b95f89f9fff498e7a" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39695 } } peers { permanent_uuid: "c241085ff3bf4461b00a253bb9627f6b" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 34817 } } peers { permanent_uuid: "021b306ba910400ab06428ccb5c55861" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35833 } }
I20260929 10:29:54.436977 32569 ts_tablet_manager.cc:1428] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 10:29:54.437335 32549 heartbeater.cc:503] Master 127.20.92.62:35923 was elected leader, sending a full tablet report...
I20260929 10:29:54.466434 32575 raft_consensus.cc:482] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:29:54.466699 32575 raft_consensus.cc:504] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaad6afddabf465b95f89f9fff498e7a" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39695 } } peers { permanent_uuid: "c241085ff3bf4461b00a253bb9627f6b" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 34817 } } peers { permanent_uuid: "021b306ba910400ab06428ccb5c55861" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35833 } }
I20260929 10:29:54.468091 32575 leader_election.cc:288] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaad6afddabf465b95f89f9fff498e7a (127.20.92.2:39695), 021b306ba910400ab06428ccb5c55861 (127.20.92.1:35833)
I20260929 10:29:54.472772 32551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:54.468289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:39695 (local address 127.20.92.3:51469)
0929 10:29:54.468416 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 10:29:54.468435 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 10:29:54.468534 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:29:54.468901 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:29:54.468905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:29:54.468923 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:29:54.469046 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:54.469052 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:54.470049 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:54.470055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:54.471297 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:54.471307 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:54.472276 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:54.472279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:54.472469 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:54.472491 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 10:29:54.472556 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 10:29:54.472627 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":46}
I20260929 10:29:54.473116 32562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:54.468575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:51469 (local address 127.20.92.2:39695)
0929 10:29:54.468698 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:54.468702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:54.468712 (+ 10us) server_negotiation.cc:407] Connection header received
0929 10:29:54.468762 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:54.468764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:54.468799 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:54.468888 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:29:54.469172 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:54.469915 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:54.471451 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:54.472051 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:54.472149 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:54.472211 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 10:29:54.472922 (+ 711us) server_negotiation.cc:299] Negotiation successful
0929 10:29:54.473006 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 10:29:54.474465 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1719b3f186b44dc5b442cd1264cec9e8" candidate_uuid: "c241085ff3bf4461b00a253bb9627f6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaad6afddabf465b95f89f9fff498e7a" is_pre_election: true
I20260929 10:29:54.474807 32360 raft_consensus.cc:2454] T 1719b3f186b44dc5b442cd1264cec9e8 P eaad6afddabf465b95f89f9fff498e7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c241085ff3bf4461b00a253bb9627f6b in term 0.
I20260929 10:29:54.475260 32436 leader_election.cc:302] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c241085ff3bf4461b00a253bb9627f6b, eaad6afddabf465b95f89f9fff498e7a; no voters:
I20260929 10:29:54.475528 32575 raft_consensus.cc:2781] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:29:54.475628 32575 raft_consensus.cc:482] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:29:54.475687 32575 raft_consensus.cc:3037] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:29:54.476325 32566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:54.469651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:36505 (local address 127.20.92.1:35833)
0929 10:29:54.469796 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:54.469801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:54.470072 (+ 271us) server_negotiation.cc:407] Connection header received
0929 10:29:54.470241 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:54.470244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:54.470295 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:54.470342 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:29:54.470701 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:54.471412 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:54.472831 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:54.473395 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:54.473457 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:54.473500 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 10:29:54.476125 (+ 2625us) server_negotiation.cc:299] Negotiation successful
0929 10:29:54.476207 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 10:29:54.476848 32576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:54.469290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:35833 (local address 127.20.92.3:36505)
0929 10:29:54.469996 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0929 10:29:54.470011 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:29:54.470132 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:29:54.470467 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:29:54.470469 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:29:54.470485 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:29:54.470584 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:54.470589 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:54.471533 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:54.471536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:54.472693 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:29:54.472701 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:29:54.475837 (+ 3136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:29:54.475842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:29:54.475967 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:54.475996 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 10:29:54.476593 (+ 597us) client_negotiation.cc:240] Negotiation successful
0929 10:29:54.476668 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":50,"thread_start_us":109,"threads_started":1}
I20260929 10:29:54.477313 32575 raft_consensus.cc:504] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaad6afddabf465b95f89f9fff498e7a" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39695 } } peers { permanent_uuid: "c241085ff3bf4461b00a253bb9627f6b" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 34817 } } peers { permanent_uuid: "021b306ba910400ab06428ccb5c55861" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35833 } }
I20260929 10:29:54.477520 32233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1719b3f186b44dc5b442cd1264cec9e8" candidate_uuid: "c241085ff3bf4461b00a253bb9627f6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021b306ba910400ab06428ccb5c55861" is_pre_election: true
I20260929 10:29:54.477734 32575 leader_election.cc:288] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b [CANDIDATE]: Term 1 election: Requested vote from peers eaad6afddabf465b95f89f9fff498e7a (127.20.92.2:39695), 021b306ba910400ab06428ccb5c55861 (127.20.92.1:35833)
I20260929 10:29:54.478183 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1719b3f186b44dc5b442cd1264cec9e8" candidate_uuid: "c241085ff3bf4461b00a253bb9627f6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaad6afddabf465b95f89f9fff498e7a"
I20260929 10:29:54.478281 32232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1719b3f186b44dc5b442cd1264cec9e8" candidate_uuid: "c241085ff3bf4461b00a253bb9627f6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021b306ba910400ab06428ccb5c55861"
I20260929 10:29:54.478307 32360 raft_consensus.cc:3037] T 1719b3f186b44dc5b442cd1264cec9e8 P eaad6afddabf465b95f89f9fff498e7a [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:29:54.479446 32360 raft_consensus.cc:2454] T 1719b3f186b44dc5b442cd1264cec9e8 P eaad6afddabf465b95f89f9fff498e7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c241085ff3bf4461b00a253bb9627f6b in term 1.
I20260929 10:29:54.479797 32233 raft_consensus.cc:2454] T 1719b3f186b44dc5b442cd1264cec9e8 P 021b306ba910400ab06428ccb5c55861 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c241085ff3bf4461b00a253bb9627f6b in term 0.
I20260929 10:29:54.479804 32436 leader_election.cc:302] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c241085ff3bf4461b00a253bb9627f6b, eaad6afddabf465b95f89f9fff498e7a; no voters: 021b306ba910400ab06428ccb5c55861
I20260929 10:29:54.480188 32575 raft_consensus.cc:2781] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:29:54.480484 32575 raft_consensus.cc:686] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b [term 1 LEADER]: Becoming Leader. State: Replica: c241085ff3bf4461b00a253bb9627f6b, State: Running, Role: LEADER
I20260929 10:29:54.482203 32575 consensus_queue.cc:237] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaad6afddabf465b95f89f9fff498e7a" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39695 } } peers { permanent_uuid: "c241085ff3bf4461b00a253bb9627f6b" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 34817 } } peers { permanent_uuid: "021b306ba910400ab06428ccb5c55861" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35833 } }
I20260929 10:29:54.485855 32064 catalog_manager.cc:5495] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b reported cstate change: term changed from 0 to 1, leader changed from <none> to c241085ff3bf4461b00a253bb9627f6b (127.20.92.3). New cstate: current_term: 1 leader_uuid: "c241085ff3bf4461b00a253bb9627f6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaad6afddabf465b95f89f9fff498e7a" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c241085ff3bf4461b00a253bb9627f6b" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 34817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021b306ba910400ab06428ccb5c55861" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35833 } health_report { overall_health: UNKNOWN } } }
I20260929 10:29:54.508122 32565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:54.504305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49100 (local address 127.20.92.3:34817)
0929 10:29:54.504432 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 10:29:54.504438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:29:54.504505 (+ 67us) server_negotiation.cc:407] Connection header received
0929 10:29:54.504671 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:29:54.504673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:29:54.504728 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:29:54.504852 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:29:54.505119 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:54.505709 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:54.506666 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:29:54.506890 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:29:54.507002 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:29:54.507328 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:29:54.507739 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:29:54.507879 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 10:29:54.507920 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 10:29:54.507975 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 10:29:54.513613 32233 raft_consensus.cc:3037] T 1719b3f186b44dc5b442cd1264cec9e8 P 021b306ba910400ab06428ccb5c55861 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:29:54.513833 32360 raft_consensus.cc:1261] T 1719b3f186b44dc5b442cd1264cec9e8 P eaad6afddabf465b95f89f9fff498e7a [term 1 FOLLOWER]: Refusing update from remote peer c241085ff3bf4461b00a253bb9627f6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:29:54.514569 32575 consensus_queue.cc:1035] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaad6afddabf465b95f89f9fff498e7a" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:29:54.515223 32233 raft_consensus.cc:1261] T 1719b3f186b44dc5b442cd1264cec9e8 P 021b306ba910400ab06428ccb5c55861 [term 1 FOLLOWER]: Refusing update from remote peer c241085ff3bf4461b00a253bb9627f6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:29:54.516052 32577 consensus_queue.cc:1035] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "021b306ba910400ab06428ccb5c55861" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 35833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:29:54.529590 32583 mvcc.cc:205] Tried to move back new op lower bound from 7334616246321184768 to 7334616246201962496. Current Snapshot: MvccSnapshot[applied={T|T < 7334616246321184768}]
I20260929 10:29:54.541180 32064 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:29:54.541397 32064 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:29:54.544268 32064 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09d99dd4320544df97e466286f4fd557: Sending DeleteTablet for 3 replicas of tablet 1719b3f186b44dc5b442cd1264cec9e8
I20260929 10:29:54.545229 32346 tablet_service.cc:1514] Processing DeleteTablet for tablet 1719b3f186b44dc5b442cd1264cec9e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:29:54 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:33502
I20260929 10:29:54.545562 32464 tablet_service.cc:1514] Processing DeleteTablet for tablet 1719b3f186b44dc5b442cd1264cec9e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:29:54 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:49098
I20260929 10:29:54.545948 32203 tablet_service.cc:1514] Processing DeleteTablet for tablet 1719b3f186b44dc5b442cd1264cec9e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:29:54 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:37870
I20260929 10:29:54.546804 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 32146
I20260929 10:29:54.549938 32590 tablet_replica.cc:330] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b: stopping tablet replica
I20260929 10:29:54.550274 32590 raft_consensus.cc:2229] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:29:54.550699 32590 raft_consensus.cc:2258] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:29:54.553057 32590 ts_tablet_manager.cc:1905] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:29:54.553968 32589 tablet_replica.cc:330] T 1719b3f186b44dc5b442cd1264cec9e8 P eaad6afddabf465b95f89f9fff498e7a: stopping tablet replica
I20260929 10:29:54.554311 32589 raft_consensus.cc:2229] T 1719b3f186b44dc5b442cd1264cec9e8 P eaad6afddabf465b95f89f9fff498e7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:29:54.554631 32589 raft_consensus.cc:2258] T 1719b3f186b44dc5b442cd1264cec9e8 P eaad6afddabf465b95f89f9fff498e7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:29:54.556263 32589 ts_tablet_manager.cc:1905] T 1719b3f186b44dc5b442cd1264cec9e8 P eaad6afddabf465b95f89f9fff498e7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:29:54.556439 32590 ts_tablet_manager.cc:1918] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:29:54.556552 32590 log.cc:1196] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-2/wal/wals/1719b3f186b44dc5b442cd1264cec9e8
I20260929 10:29:54.556902 32590 ts_tablet_manager.cc:1939] T 1719b3f186b44dc5b442cd1264cec9e8 P c241085ff3bf4461b00a253bb9627f6b: Deleting consensus metadata
I20260929 10:29:54.558652 32051 catalog_manager.cc:4841] TS c241085ff3bf4461b00a253bb9627f6b (127.20.92.3:34817): tablet 1719b3f186b44dc5b442cd1264cec9e8 (table test-table [id=09d6f0824d244822ad4ce2b8863afcf9]) successfully deleted
I20260929 10:29:54.558815 32589 ts_tablet_manager.cc:1918] T 1719b3f186b44dc5b442cd1264cec9e8 P eaad6afddabf465b95f89f9fff498e7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:29:54.558911 32589 log.cc:1196] T 1719b3f186b44dc5b442cd1264cec9e8 P eaad6afddabf465b95f89f9fff498e7a: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677679287822-20848-0/minicluster-data/ts-1/wal/wals/1719b3f186b44dc5b442cd1264cec9e8
I20260929 10:29:54.559224 32589 ts_tablet_manager.cc:1939] T 1719b3f186b44dc5b442cd1264cec9e8 P eaad6afddabf465b95f89f9fff498e7a: Deleting consensus metadata
I20260929 10:29:54.560401 32049 catalog_manager.cc:4841] TS eaad6afddabf465b95f89f9fff498e7a (127.20.92.2:39695): tablet 1719b3f186b44dc5b442cd1264cec9e8 (table test-table [id=09d6f0824d244822ad4ce2b8863afcf9]) successfully deleted
W20260929 10:29:54.561364 32052 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.92.1:35833 (error 108)
I20260929 10:29:54.562325 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 32285
I20260929 10:29:54.563067 32564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:29:54.562118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:35833 (local address 127.0.0.1:37886)
0929 10:29:54.562864 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0929 10:29:54.562957 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.1:35833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":625}
W20260929 10:29:54.563321 32052 catalog_manager.cc:4584] TS 021b306ba910400ab06428ccb5c55861 (127.20.92.1:35833): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1719b3f186b44dc5b442cd1264cec9e8: Network error: Client connection negotiation failed: client connection to 127.20.92.1:35833: connect: Connection refused (error 111)
I20260929 10:29:54.577215 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 32420
I20260929 10:29:54.583796 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 32033
I20260929 10:29:54.589795 20848 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 10:29:54.925015 20848 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 10:29:54.925155 20848 mini_ranger.cc:66] Stopping Ranger...
I20260929 10:29:55.271116 20848 mini_ranger.cc:68] Stopped Ranger
2026-09-29 10:29:55.271 UTC [31619] LOG: received smart shutdown request
2026-09-29 10:29:55.275 UTC [31619] LOG: background worker "logical replication launcher" (PID 31628) exited with exit code 1
2026-09-29 10:29:55.276 UTC [31623] LOG: shutting down
2026-09-29 10:29:55.531 UTC [31619] LOG: database system is shut down
2026-09-29T10:29:55Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50973 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:29:55 dist-test-slave-9cj5 krb5kdc[32602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:29:55 dist-test-slave-9cj5 krb5kdc[32602](info): set up 2 sockets
Sep 29 10:29:55 dist-test-slave-9cj5 krb5kdc[32602](info): commencing operation
krb5kdc: starting...
W20260929 10:29:56.754668 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:29:56 dist-test-slave-9cj5 krb5kdc[32602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:29:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:29:56Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.20.92.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.20.92.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.20.92.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/rangeradmin_127.20.92.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.20.92.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/rangeradmin_127.20.92.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.20.92.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.20.92.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.20.92.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/rangerlookup_127.20.92.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.20.92.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/rangerlookup_127.20.92.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.92.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.92.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.92.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/HTTP_127.20.92.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.92.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/HTTP_127.20.92.20@KRBTEST.COM.keytab.
I20260929 10:29:56.961406 20848 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task8Q3Hzv/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/postgres -l logfile start
2026-09-29 10:30:00.133 UTC [32632] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 10:30:00.133 UTC [32632] LOG: listening on IPv4 address "127.20.92.20", port 59787
2026-09-29 10:30:00.138 UTC [32632] LOG: listening on Unix socket "/tmp/.s.PGSQL.59787"
2026-09-29 10:30:00.176 UTC [32635] LOG: database system was shut down at 2026-09-29 10:29:58 UTC
2026-09-29 10:30:00.182 UTC [32632] LOG: database system is ready to accept connections
I20260929 10:30:01.162482 20848 mini_postgres.cc:96] Postgres bound to 59787
2026-09-29 10:30:01.169 UTC [32643] FATAL: database "slave" does not exist
127.20.92.20:59787 - accepting connections
I20260929 10:30:01.169888 20848 mini_ranger.cc:161] Starting Ranger...
I20260929 10:30:01.187253 20848 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 10:30:01.925272 20848 mini_ranger.cc:87] Created ranger Postgres database
I20260929 10:30:01.925374 20848 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ranger-admin
2026-09-29 10:30:02,262 [I] DB FLAVOR :POSTGRES
2026-09-29 10:30:02,263 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 10:30:02,263 [I] Checking connection..
2026-09-29 10:30:02,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:59787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 10:30:02,598 [I] Checking connection passed.
2026-09-29 10:30:02,599 [I] --------- Verifying version history table ---------
2026-09-29 10:30:02,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:59787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 10:30:02,910 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 10:30:02,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:59787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 10:30:03,207 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 10:30:03,207 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 10:30:03,208 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:59787/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 10:30:03.516 UTC [32707] WARNING: there is no transaction in progress
2026-09-29 10:30:03,524 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 10:30:03,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:59787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 10:30:03,824 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 10:30:03,825 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 10:30:03,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:59787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 10:30:04,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:59787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 10:30:04,490 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:59787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9cj5.c.gcp-upstream.internal','N') ;"
2026-09-29 10:30:04,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:59787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 10:30:05,094 [I] Table x_portal_user does not exist in database ranger
2026-09-29 10:30:05,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:59787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 10:30:05,424 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 10:30:05,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:59787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 10:30:05,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:59787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 10:30:05,996 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 10:30:05,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:59787/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 10:30:07.517 UTC [347] WARNING: there is no transaction in progress
2026-09-29 10:30:07.547 UTC [347] WARNING: there is no transaction in progress
2026-09-29 10:30:07.584 UTC [347] WARNING: there is no transaction in progress
2026-09-29 10:30:07.609 UTC [347] WARNING: there is no transaction in progress
2026-09-29 10:30:07.634 UTC [347] WARNING: there is no transaction in progress
2026-09-29 10:30:07.659 UTC [347] WARNING: there is no transaction in progress
2026-09-29 10:30:07.747 UTC [347] WARNING: there is no transaction in progress
2026-09-29 10:30:07.758 UTC [347] WARNING: there is no transaction in progress
2026-09-29 10:30:07.785 UTC [347] WARNING: there is no transaction in progress
2026-09-29 10:30:07.801 UTC [347] WARNING: there is no transaction in progress
2026-09-29 10:30:07.820 UTC [347] WARNING: there is no transaction in progress
2026-09-29 10:30:07.845 UTC [347] WARNING: there is no transaction in progress
2026-09-29 10:30:07.864 UTC [347] WARNING: there is no transaction in progress
2026-09-29 10:30:07.878 UTC [347] WARNING: there is no transaction in progress
2026-09-29 10:30:07.892 UTC [347] WARNING: there is no transaction in progress
2026-09-29 10:30:07.913 UTC [347] WARNING: there is no transaction in progress
2026-09-29 10:30:09,303 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 10:30:09,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:59787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 10:30:09,623 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 10:30:09,623 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:59787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 10:30:09,928 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 10:30:09,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:59787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 10:30:10,233 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 10:30:10,234 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:59787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 10:30:10,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:59787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 10:30:10,833 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:59787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9cj5.c.gcp-upstream.internal';"
2026-09-29 10:30:11,133 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 10:30:11,133 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:59787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 10:30:11,444 [I] DB_PATCHES have already been applied
I20260929 10:30:11.451740 20848 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ranger-admin:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task8Q3Hzv/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 10:30:11.451817 20848 mini_ranger.cc:193] Using host: 127.20.92.20
I20260929 10:30:11.455770 20848 mini_ranger.cc:227] Ranger admin URL: http://127.20.92.20:58229
Sep 29, 2026 10:30:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 10:30:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 10:30:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 10:30:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 10:30:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 10:30:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 10:30:12 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 10:30:12.741609 20848 mini_ranger.cc:160] Time spent starting Ranger: real 11.572s user 0.003s sys 0.009s
Sep 29, 2026 10:30:34 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 10:30:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 10:30:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 10:30:34 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 10:30:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 10:30:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 10:30:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 10:30:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 10:30:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 10:30:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 10:30:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 10:30:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 10:30:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 10:30:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 10:30:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 10:30:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 10:30:34 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 10:30:35 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 10:30:35.953455 20848 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.20.92.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.20.92.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.20.92.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/rangerkms_127.20.92.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.20.92.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/rangerkms_127.20.92.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.92.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.92.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.92.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/HTTP_127.20.92.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.92.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/HTTP_127.20.92.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 10:30:36 dist-test-slave-9cj5 krb5kdc[32602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677836, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 10:30:36.045660 20848 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 10:30:36.056731 20848 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 10:30:37.346541 20848 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 10:30:37.346617 20848 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ranger-kms
I20260929 10:30:37.346630 20848 mini_ranger_kms.cc:227] Using postgres at 127.20.92.20:59787
2026-09-29 10:30:37,522 [I] DB FLAVOR :POSTGRES
2026-09-29 10:30:37,523 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 10:30:37,523 [I] Checking connection
2026-09-29 10:30:37,523 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:59787/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 10:30:37,788 [I] connection success
2026-09-29 10:30:37,789 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 10:30:37,789 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 10:30:37,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:59787/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 10:30:38,058 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 10:30:38,058 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 10:30:38,059 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 10:30:38,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.92.20:59787/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 10:30:38,345 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 10:30:38.886803 20848 mini_ranger_kms.cc:310] Created kms service
I20260929 10:30:39.115792 20848 mini_ranger_kms.cc:326] Created kudu user
I20260929 10:30:39.172868 20848 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 10:30:39 dist-test-slave-9cj5 krb5kdc[32602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677836, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.20.92.20@KRBTEST.COM
I20260929 10:30:39.632020 20848 mini_ranger_kms.cc:384] Added ranger policy
I20260929 10:30:39.632290 20848 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ranger-kms:/tmp/dist-test-task8Q3Hzv/build/debug/bin/postgresql.jar:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task8Q3Hzv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 10:30:39.632320 20848 mini_ranger_kms.cc:240] Using host: 127.20.92.20
I20260929 10:30:39.636234 20848 mini_ranger_kms.cc:276] Ranger KMS PID: 528
I20260929 10:30:39.636324 20848 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.20.92.20:39913
Sep 29, 2026 10:30:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 10:30:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 10:30:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 10:30:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 10:30:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 10:30:41 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 29 10:30:41 dist-test-slave-9cj5 krb5kdc[32602](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790677841, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.92.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:30:42.039491 20848 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.994s user 0.006s sys 0.152s
I20260929 10:30:42.039623 20848 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 10:30:42.039688 20848 mini_ranger_kms.cc:401] 127.20.92.20:39913/kms/v1/keys
Sep 29 10:30:42 dist-test-slave-9cj5 krb5kdc[32602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677841, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.92.20@KRBTEST.COM for HTTP/127.20.92.20@KRBTEST.COM
Sep 29 10:30:42 dist-test-slave-9cj5 krb5kdc[32602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677841, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.92.20@KRBTEST.COM for HTTP/127.20.92.20@KRBTEST.COM
Sep 29 10:30:45 dist-test-slave-9cj5 krb5kdc[32602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.20.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
I20260929 10:30:45.323488 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:37361
--webserver_interface=127.20.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:37361
--ranger_config_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.92.20:39913/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:30:45.463130 578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:30:45.463464 578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:30:45.463506 578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:30:45.468369 578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:30:45.468441 578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:30:45.468461 578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:30:45.468477 578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:30:45.468493 578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:30:45.474503 578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/wal
--ranger_kms_url=127.20.92.20:39913/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:37361
--ranger_config_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:37361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:30:45.475584 578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:30:45.476568 578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:30:45.487020 584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:45.487095 586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:45.487046 578 server_base.cc:1034] running on GCE node
W20260929 10:30:45.487067 583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:45.487926 578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:30:45.489132 578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:30:45.490356 578 hybrid_clock.cc:648] HybridClock initialized: now 1790677845490325 us; error 44 us; skew 500 ppm
Sep 29 10:30:45 dist-test-slave-9cj5 krb5kdc[32602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677845, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:30:45.496949 578 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:30:45.498005 578 webserver.cc:460] Webserver started at http://127.20.92.62:33713/ using document root <none> and password file <none>
I20260929 10:30:45.498564 578 fs_manager.cc:359] Metadata directory not provided
I20260929 10:30:45.498620 578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:30:45.498797 578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 10:30:45 dist-test-slave-9cj5 krb5kdc[32602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677845, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for HTTP/127.20.92.20@KRBTEST.COM
I20260929 10:30:45.564713 578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "0888c6d1da944f7eb019027bade5962c"
format_stamp: "Formatted at 2026-09-29 10:30:45 on dist-test-slave-9cj5"
server_key: "f98dd5a98c5243d1364bc7185f5d798b"
server_key_iv: "f180116ace8e478a37460bb0efb1dc35"
server_key_version: "kuduclusterkey@0"
I20260929 10:30:45.565510 578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "0888c6d1da944f7eb019027bade5962c"
format_stamp: "Formatted at 2026-09-29 10:30:45 on dist-test-slave-9cj5"
server_key: "f98dd5a98c5243d1364bc7185f5d798b"
server_key_iv: "f180116ace8e478a37460bb0efb1dc35"
server_key_version: "kuduclusterkey@0"
I20260929 10:30:45.570701 578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 10:30:45.590157 593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:45.591720 578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 10:30:45.591895 578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "0888c6d1da944f7eb019027bade5962c"
format_stamp: "Formatted at 2026-09-29 10:30:45 on dist-test-slave-9cj5"
server_key: "f98dd5a98c5243d1364bc7185f5d798b"
server_key_iv: "f180116ace8e478a37460bb0efb1dc35"
server_key_version: "kuduclusterkey@0"
I20260929 10:30:45.592041 578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:30:45.604200 578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:30:45.607667 578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:30:45.607867 578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:30:45.617071 578 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:37361
I20260929 10:30:45.617084 645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:37361 every 8 connection(s)
I20260929 10:30:45.618430 578 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:30:45.621353 646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:30:45.623489 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 578
I20260929 10:30:45.623633 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:30:45.623935 20848 external_mini_cluster.cc:1428] Setting key d3a7ff83a67869fb1c61ed32757753a1
I20260929 10:30:45.628927 646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c: Bootstrap starting.
I20260929 10:30:45.631556 646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c: Neither blocks nor log segments found. Creating new log.
I20260929 10:30:45.632478 646 log.cc:824] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c: Log is configured to *not* fsync() on all Append() calls
Sep 29 10:30:45 dist-test-slave-9cj5 krb5kdc[32602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:30:45.634778 646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c: No bootstrap required, opened a new log
I20260929 10:30:45.638141 646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0888c6d1da944f7eb019027bade5962c" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37361 } }
I20260929 10:30:45.638396 646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:30:45.638466 646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0888c6d1da944f7eb019027bade5962c, State: Initialized, Role: FOLLOWER
I20260929 10:30:45.638993 646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0888c6d1da944f7eb019027bade5962c" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37361 } }
I20260929 10:30:45.639159 646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:30:45.639247 646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:30:45.639364 646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:30:45.639676 649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:45.625483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58402 (local address 127.20.92.62:37361)
0929 10:30:45.625970 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:45.625990 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:45.626037 (+ 47us) server_negotiation.cc:407] Connection header received
0929 10:30:45.626968 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:45.626993 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:45.627527 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:45.627917 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:45.628831 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:45.630100 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:45.631266 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:45.631868 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:45.631997 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:45.634892 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:45.634899 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:45.634919 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:45.634958 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:45.636643 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:45.637116 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:45.637121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:45.637126 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:45.637230 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:45.637521 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:45.637525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:45.637527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:45.637998 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:45.638220 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:45.638483 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 10:30:45.638802 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":122,"threads_started":1}
I20260929 10:30:45.640592 646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0888c6d1da944f7eb019027bade5962c" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37361 } }
I20260929 10:30:45.641062 646 leader_election.cc:302] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0888c6d1da944f7eb019027bade5962c; no voters:
I20260929 10:30:45.641527 646 leader_election.cc:288] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:30:45.641635 651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:30:45.641918 651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [term 1 LEADER]: Becoming Leader. State: Replica: 0888c6d1da944f7eb019027bade5962c, State: Running, Role: LEADER
I20260929 10:30:45.642354 651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0888c6d1da944f7eb019027bade5962c" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37361 } }
I20260929 10:30:45.642858 646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:30:45.643993 653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0888c6d1da944f7eb019027bade5962c. Latest consensus state: current_term: 1 leader_uuid: "0888c6d1da944f7eb019027bade5962c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0888c6d1da944f7eb019027bade5962c" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37361 } } }
I20260929 10:30:45.644121 653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [sys.catalog]: This master's current role is: LEADER
I20260929 10:30:45.644421 652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0888c6d1da944f7eb019027bade5962c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0888c6d1da944f7eb019027bade5962c" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37361 } } }
I20260929 10:30:45.644500 652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [sys.catalog]: This master's current role is: LEADER
I20260929 10:30:45.644528 660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:30:45.647816 660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:30:45.648474 646 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 10:30:45.658522 660 catalog_manager.cc:1347] Generated new cluster ID: 648181a861e44b76aef00def809ea2f7
I20260929 10:30:45.658612 660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:30:45.673533 660 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:30:45.675144 660 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:30:45.692268 660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c: Generated new TSK 0
I20260929 10:30:45.693454 660 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
I20260929 10:30:46.439476 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.92.20:39913/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:0
--local_ip_for_outbound_sockets=127.20.92.1
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:37361
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:30:46.603678 691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:30:46.604079 691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:30:46.604146 691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:30:46.612833 691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:30:46.612936 691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:30:46.613021 691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:30:46.622030 691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.20.92.20:39913/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:37361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:30:46.624473 691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:30:46.626185 691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:30:46.640187 699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:46.640368 698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:46.640887 691 server_base.cc:1034] running on GCE node
W20260929 10:30:46.640916 701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:46.641433 691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:30:46.642377 691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:30:46.643599 691 hybrid_clock.cc:648] HybridClock initialized: now 1790677846643562 us; error 60 us; skew 500 ppm
Sep 29 10:30:46 dist-test-slave-9cj5 krb5kdc[32602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677846, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:30:46.648011 691 init.cc:377] Logged in from keytab as kudu/127.20.92.1@KRBTEST.COM (short username kudu)
I20260929 10:30:46.649406 691 webserver.cc:460] Webserver started at http://127.20.92.1:46581/ using document root <none> and password file <none>
I20260929 10:30:46.650312 691 fs_manager.cc:359] Metadata directory not provided
I20260929 10:30:46.650432 691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:30:46.650816 691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 10:30:46 dist-test-slave-9cj5 krb5kdc[32602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677846, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for HTTP/127.20.92.20@KRBTEST.COM
I20260929 10:30:46.673554 691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-0/data/instance:
uuid: "243e4c62a44e454782fc97ee4831c3af"
format_stamp: "Formatted at 2026-09-29 10:30:46 on dist-test-slave-9cj5"
server_key: "258be2483e2a0860c3d66d7645493e83"
server_key_iv: "5cdee28a09079cf9f8cf2911637f68e2"
server_key_version: "kuduclusterkey@0"
I20260929 10:30:46.674865 691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance:
uuid: "243e4c62a44e454782fc97ee4831c3af"
format_stamp: "Formatted at 2026-09-29 10:30:46 on dist-test-slave-9cj5"
server_key: "258be2483e2a0860c3d66d7645493e83"
server_key_iv: "5cdee28a09079cf9f8cf2911637f68e2"
server_key_version: "kuduclusterkey@0"
I20260929 10:30:46.681488 691 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.003s
I20260929 10:30:46.710460 708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:46.713094 691 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.001s
I20260929 10:30:46.713421 691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "243e4c62a44e454782fc97ee4831c3af"
format_stamp: "Formatted at 2026-09-29 10:30:46 on dist-test-slave-9cj5"
server_key: "258be2483e2a0860c3d66d7645493e83"
server_key_iv: "5cdee28a09079cf9f8cf2911637f68e2"
server_key_version: "kuduclusterkey@0"
I20260929 10:30:46.713591 691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:30:46.743189 691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:30:46.748013 691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:30:46.748273 691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:30:46.749051 691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:30:46.750357 691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:30:46.750483 691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 10:30:46.750612 691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:30:46.750691 691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:46.765357 691 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:34643
I20260929 10:30:46.768112 691 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
I20260929 10:30:46.771553 823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:34643 every 8 connection(s)
I20260929 10:30:46.774941 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 691
I20260929 10:30:46.775089 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance
I20260929 10:30:46.775398 20848 external_mini_cluster.cc:1428] Setting key 0fa1c8621400224ae9fc475c6f6314a9
Sep 29 10:30:46 dist-test-slave-9cj5 krb5kdc[32602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677846, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:30:46.793174 827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:46.774305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:37361 (local address 127.20.92.1:40147)
0929 10:30:46.776149 (+ 1844us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:46.776273 (+ 124us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:46.777432 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:46.778541 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:46.778553 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:46.778967 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:30:46.779727 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:46.779750 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:46.781086 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:46.781090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:46.782773 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:46.782788 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:46.783693 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:46.783698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:46.783937 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:46.784783 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:30:46.784809 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:30:46.786847 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:30:46.789572 (+ 2725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:46.789580 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:46.789602 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:46.790039 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:46.790395 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:46.790398 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:46.790400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:46.790601 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:46.791089 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:30:46.791097 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:30:46.791486 (+ 389us) client_negotiation.cc:769] Sending connection context
0929 10:30:46.791695 (+ 209us) client_negotiation.cc:240] Negotiation successful
0929 10:30:46.792064 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1594,"thread_start_us":167,"threads_started":1}
I20260929 10:30:46.794242 826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:46.773698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:40147 (local address 127.20.92.62:37361)
0929 10:30:46.777949 (+ 4251us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:46.777954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:46.777970 (+ 16us) server_negotiation.cc:407] Connection header received
0929 10:30:46.778018 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:46.778021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:46.778075 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:46.778185 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:46.779932 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:46.780908 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:46.782959 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:46.783355 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:46.783473 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:46.787112 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:46.787119 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:46.787125 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:46.787176 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:46.789396 (+ 2220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:46.790186 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:46.790190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:46.790192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:46.790291 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:46.790715 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:46.790718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:46.790721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:46.790954 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:46.791038 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:46.793833 (+ 2795us) server_negotiation.cc:299] Negotiation successful
0929 10:30:46.794016 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4158,"thread_start_us":116,"threads_started":1}
I20260929 10:30:46.795683 824 heartbeater.cc:347] Connected to a master server at 127.20.92.62:37361
I20260929 10:30:46.795953 824 heartbeater.cc:464] Registering TS with master...
I20260929 10:30:46.796651 824 heartbeater.cc:511] Master 127.20.92.62:37361 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.92.2@KRBTEST.COM; defaulting to no policy
I20260929 10:30:46.805825 610 ts_manager.cc:194] Registered new tserver with Master: 243e4c62a44e454782fc97ee4831c3af (127.20.92.1:34643)
I20260929 10:30:46.807633 610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.1@KRBTEST.COM'} at 127.20.92.1:40147
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
I20260929 10:30:46.847534 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.92.20:39913/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:0
--local_ip_for_outbound_sockets=127.20.92.2
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:37361
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 10:30:46 dist-test-slave-9cj5 krb5kdc[32602](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790677846, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 10:30:47.035059 830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:30:47.035456 830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:30:47.035524 830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:30:47.042475 830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:30:47.042562 830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:30:47.042654 830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:30:47.051545 830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.20.92.20:39913/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:37361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:30:47.053267 830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:30:47.054786 830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:30:47.070915 837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:47.071027 836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:47.071815 839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:47.072232 830 server_base.cc:1034] running on GCE node
I20260929 10:30:47.073254 830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:30:47.074208 830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:30:47.077636 830 hybrid_clock.cc:648] HybridClock initialized: now 1790677847077539 us; error 99 us; skew 500 ppm
Sep 29 10:30:47 dist-test-slave-9cj5 krb5kdc[32602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677847, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:30:47.082697 830 init.cc:377] Logged in from keytab as kudu/127.20.92.2@KRBTEST.COM (short username kudu)
I20260929 10:30:47.084066 830 webserver.cc:460] Webserver started at http://127.20.92.2:33519/ using document root <none> and password file <none>
I20260929 10:30:47.084900 830 fs_manager.cc:359] Metadata directory not provided
I20260929 10:30:47.085070 830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:30:47.085392 830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 10:30:47 dist-test-slave-9cj5 krb5kdc[32602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677847, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for HTTP/127.20.92.20@KRBTEST.COM
I20260929 10:30:47.113976 830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-1/data/instance:
uuid: "f7e3a29a14394eb0a11aaf06d4cec6e4"
format_stamp: "Formatted at 2026-09-29 10:30:47 on dist-test-slave-9cj5"
server_key: "9227ade540e6fb8f765c0667e6c10bd4"
server_key_iv: "8bd348aae9d848bf9d5815c98ec8bf74"
server_key_version: "kuduclusterkey@0"
I20260929 10:30:47.115199 830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance:
uuid: "f7e3a29a14394eb0a11aaf06d4cec6e4"
format_stamp: "Formatted at 2026-09-29 10:30:47 on dist-test-slave-9cj5"
server_key: "9227ade540e6fb8f765c0667e6c10bd4"
server_key_iv: "8bd348aae9d848bf9d5815c98ec8bf74"
server_key_version: "kuduclusterkey@0"
I20260929 10:30:47.120693 830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 10:30:47.155191 846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:47.157174 830 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.000s
I20260929 10:30:47.157354 830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "f7e3a29a14394eb0a11aaf06d4cec6e4"
format_stamp: "Formatted at 2026-09-29 10:30:47 on dist-test-slave-9cj5"
server_key: "9227ade540e6fb8f765c0667e6c10bd4"
server_key_iv: "8bd348aae9d848bf9d5815c98ec8bf74"
server_key_version: "kuduclusterkey@0"
I20260929 10:30:47.157511 830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:30:47.188031 830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:30:47.192577 830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:30:47.192819 830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:30:47.193589 830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:30:47.194911 830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:30:47.195035 830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:47.195163 830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:30:47.195236 830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:47.208112 830 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:39887
I20260929 10:30:47.208129 959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:39887 every 8 connection(s)
I20260929 10:30:47.209497 830 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
I20260929 10:30:47.212572 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 830
I20260929 10:30:47.212729 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance
I20260929 10:30:47.213086 20848 external_mini_cluster.cc:1428] Setting key b80d87cf6accd1a55c762c4dcceb21fe
Sep 29 10:30:47 dist-test-slave-9cj5 krb5kdc[32602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677847, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.92.3@KRBTEST.COM; defaulting to no policy
I20260929 10:30:47.227782 826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:47.211883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:47319 (local address 127.20.92.62:37361)
0929 10:30:47.212040 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:47.212045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:47.214009 (+ 1964us) server_negotiation.cc:407] Connection header received
0929 10:30:47.215341 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:47.215345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:47.215403 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:47.215483 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:47.217161 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:47.217726 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:47.219039 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:47.219456 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:47.219587 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:47.222907 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:47.222912 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:47.222916 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:47.222957 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:47.224952 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:47.225718 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:47.225724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:47.225728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:47.225871 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:47.226302 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:47.226304 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:47.226305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:47.226592 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:47.226784 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:47.227421 (+ 637us) server_negotiation.cc:299] Negotiation successful
0929 10:30:47.227566 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 10:30:47.228721 962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:47.212367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:37361 (local address 127.20.92.2:47319)
0929 10:30:47.213856 (+ 1489us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:47.213905 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:47.214919 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:47.215746 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:47.215759 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:47.216191 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:30:47.216971 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:47.216995 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:47.217923 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:47.217926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:47.218882 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:47.218892 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:47.219658 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:47.219664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:47.219894 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:47.220694 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:30:47.220720 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:30:47.222649 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:30:47.225131 (+ 2482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:47.225139 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:47.225169 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:47.225577 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:47.225971 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:47.225975 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:47.225976 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:47.226195 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:47.226781 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:30:47.226789 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:30:47.227100 (+ 311us) client_negotiation.cc:769] Sending connection context
0929 10:30:47.227366 (+ 266us) client_negotiation.cc:240] Negotiation successful
0929 10:30:47.227725 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1188,"thread_start_us":156,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.3@KRBTEST.COM" created.
I20260929 10:30:47.230244 960 heartbeater.cc:347] Connected to a master server at 127.20.92.62:37361
I20260929 10:30:47.230548 960 heartbeater.cc:464] Registering TS with master...
I20260929 10:30:47.231267 960 heartbeater.cc:511] Master 127.20.92.62:37361 requested a full tablet report, sending...
I20260929 10:30:47.232481 610 ts_manager.cc:194] Registered new tserver with Master: f7e3a29a14394eb0a11aaf06d4cec6e4 (127.20.92.2:39887)
I20260929 10:30:47.233088 610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.2@KRBTEST.COM'} at 127.20.92.2:47319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
I20260929 10:30:47.262954 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.92.20:39913/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:0
--local_ip_for_outbound_sockets=127.20.92.3
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:37361
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:30:47.411309 965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:30:47.411806 965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:30:47.411931 965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:30:47.419873 965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:30:47.420054 965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:30:47.420207 965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:30:47.430413 965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.20.92.20:39913/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:37361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:30:47.432111 965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:30:47.433640 965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 29 10:30:47 dist-test-slave-9cj5 krb5kdc[32602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677846, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for HTTP/127.20.92.20@KRBTEST.COM
W20260929 10:30:47.447872 971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:47.448338 973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:47.449571 965 server_base.cc:1034] running on GCE node
W20260929 10:30:47.450316 970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:47.450922 965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:30:47.451763 965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:30:47.452919 965 hybrid_clock.cc:648] HybridClock initialized: now 1790677847452853 us; error 79 us; skew 500 ppm
Sep 29 10:30:47 dist-test-slave-9cj5 krb5kdc[32602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677847, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:30:47.457180 965 init.cc:377] Logged in from keytab as kudu/127.20.92.3@KRBTEST.COM (short username kudu)
I20260929 10:30:47.458452 965 webserver.cc:460] Webserver started at http://127.20.92.3:43735/ using document root <none> and password file <none>
I20260929 10:30:47.459247 965 fs_manager.cc:359] Metadata directory not provided
I20260929 10:30:47.459414 965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:30:47.459731 965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 10:30:47 dist-test-slave-9cj5 krb5kdc[32602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677847, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for HTTP/127.20.92.20@KRBTEST.COM
I20260929 10:30:47.494275 965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-2/data/instance:
uuid: "ae32ad291ff14447a2e7737392679b18"
format_stamp: "Formatted at 2026-09-29 10:30:47 on dist-test-slave-9cj5"
server_key: "1d478818a5fa16cd1dcd00d3c77cd11b"
server_key_iv: "e0340415fe3c7350253e4e270b1b5c03"
server_key_version: "kuduclusterkey@0"
I20260929 10:30:47.495630 965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance:
uuid: "ae32ad291ff14447a2e7737392679b18"
format_stamp: "Formatted at 2026-09-29 10:30:47 on dist-test-slave-9cj5"
server_key: "1d478818a5fa16cd1dcd00d3c77cd11b"
server_key_iv: "e0340415fe3c7350253e4e270b1b5c03"
server_key_version: "kuduclusterkey@0"
I20260929 10:30:47.515641 965 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.004s sys 0.000s
I20260929 10:30:47.554435 980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:47.556614 965 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.002s sys 0.000s
I20260929 10:30:47.556788 965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "ae32ad291ff14447a2e7737392679b18"
format_stamp: "Formatted at 2026-09-29 10:30:47 on dist-test-slave-9cj5"
server_key: "1d478818a5fa16cd1dcd00d3c77cd11b"
server_key_iv: "e0340415fe3c7350253e4e270b1b5c03"
server_key_version: "kuduclusterkey@0"
I20260929 10:30:47.556938 965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:30:47.603434 965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:30:47.607383 965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:30:47.607677 965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:30:47.608433 965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:30:47.609539 965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:30:47.609647 965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:47.609949 965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:30:47.610059 965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:47.624677 965 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:41725
I20260929 10:30:47.626493 965 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:30:47.627719 1093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:41725 every 8 connection(s)
I20260929 10:30:47.631089 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 965
I20260929 10:30:47.631273 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance
I20260929 10:30:47.631667 20848 external_mini_cluster.cc:1428] Setting key 376da2328fd03ce737e72af9ed56fb31
Sep 29 10:30:47 dist-test-slave-9cj5 krb5kdc[32602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677847, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:30:47.646013 826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:47.631335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:47407 (local address 127.20.92.62:37361)
0929 10:30:47.631479 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:47.631484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:47.632452 (+ 968us) server_negotiation.cc:407] Connection header received
0929 10:30:47.633736 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:47.633740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:47.633829 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:47.633880 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:47.635529 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:47.636378 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:47.638122 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:47.638478 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:47.638553 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:47.641572 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:47.641576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:47.641579 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:47.641608 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:47.643364 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:47.644044 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:47.644048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:47.644050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:47.644143 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:47.644568 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:47.644571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:47.644573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:47.644838 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:47.644912 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:47.645690 (+ 778us) server_negotiation.cc:299] Negotiation successful
0929 10:30:47.645861 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 10:30:47.647926 1096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:47.630125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:37361 (local address 127.20.92.3:47407)
0929 10:30:47.632290 (+ 2165us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:47.632352 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:47.633483 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:47.634326 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:47.634335 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:47.634695 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:30:47.635366 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:47.635383 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:47.636531 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:47.636535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:47.637986 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:47.637996 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:47.638758 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:47.638762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:47.638957 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:47.639621 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:30:47.639641 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:30:47.641367 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:30:47.643497 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:47.643504 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:47.643523 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:47.643922 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:47.644268 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:47.644270 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:47.644272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:47.644463 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:47.645011 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:30:47.645017 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:30:47.645406 (+ 389us) client_negotiation.cc:769] Sending connection context
0929 10:30:47.646465 (+ 1059us) client_negotiation.cc:240] Negotiation successful
0929 10:30:47.646829 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1918,"thread_start_us":161,"threads_started":1}
I20260929 10:30:47.649467 1094 heartbeater.cc:347] Connected to a master server at 127.20.92.62:37361
I20260929 10:30:47.649710 1094 heartbeater.cc:464] Registering TS with master...
I20260929 10:30:47.650303 1094 heartbeater.cc:511] Master 127.20.92.62:37361 requested a full tablet report, sending...
I20260929 10:30:47.651539 610 ts_manager.cc:194] Registered new tserver with Master: ae32ad291ff14447a2e7737392679b18 (127.20.92.3:41725)
I20260929 10:30:47.652304 610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.3@KRBTEST.COM'} at 127.20.92.3:47407
I20260929 10:30:47.657663 20848 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:30:47.658582 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 691
I20260929 10:30:47.667987 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 830
I20260929 10:30:47.676600 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 965
I20260929 10:30:47.687400 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 578
I20260929 10:30:47.695386 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:37361
--webserver_interface=127.20.92.62
--webserver_port=33713
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:37361
--ranger_config_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.92.20:39913/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.92.20:39913/kms with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:30:47.827142 1097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:30:47.827525 1097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:30:47.827613 1097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:30:47.832247 1097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:30:47.832377 1097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:30:47.832417 1097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:30:47.832470 1097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:30:47.832507 1097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:30:47.839572 1097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.92.20:39913/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:37361
--ranger_config_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:37361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=33713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:30:47.840817 1097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:30:47.842018 1097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:30:47.852361 1102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:47.852465 1105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:47.852427 1097 server_base.cc:1034] running on GCE node
W20260929 10:30:47.852361 1103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:47.853375 1097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:30:47.854636 1097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:30:47.855850 1097 hybrid_clock.cc:648] HybridClock initialized: now 1790677847855836 us; error 31 us; skew 500 ppm
Sep 29 10:30:47 dist-test-slave-9cj5 krb5kdc[32602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677847, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:30:47.858979 1097 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:30:47.859970 1097 webserver.cc:460] Webserver started at http://127.20.92.62:33713/ using document root <none> and password file <none>
I20260929 10:30:47.860654 1097 fs_manager.cc:359] Metadata directory not provided
I20260929 10:30:47.860746 1097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 10:30:47 dist-test-slave-9cj5 krb5kdc[32602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677847, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for HTTP/127.20.92.20@KRBTEST.COM
I20260929 10:30:47.928894 1097 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 10:30:47.932071 1113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:47.934238 1097 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 10:30:47.934432 1097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "0888c6d1da944f7eb019027bade5962c"
format_stamp: "Formatted at 2026-09-29 10:30:45 on dist-test-slave-9cj5"
server_key: "f98dd5a98c5243d1364bc7185f5d798b"
server_key_iv: "f180116ace8e478a37460bb0efb1dc35"
server_key_version: "kuduclusterkey@0"
I20260929 10:30:47.935175 1097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:30:47.957568 1097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:30:47.963421 1097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:30:47.963783 1097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:30:47.973784 1097 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:37361
I20260929 10:30:47.973785 1165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:37361 every 8 connection(s)
I20260929 10:30:47.975101 1097 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:30:47.981684 1166 sys_catalog.cc:263] Verifying existing consensus state
I20260929 10:30:47.984370 1166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c: Bootstrap starting.
I20260929 10:30:47.985271 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 1097
I20260929 10:30:47.985913 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.92.20:39913/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:34643
--local_ip_for_outbound_sockets=127.20.92.1
--tserver_master_addrs=127.20.92.62:37361
--webserver_port=46581
--webserver_interface=127.20.92.1
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.92.20:39913/kms with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:30:48.009063 1166 log.cc:824] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c: Log is configured to *not* fsync() on all Append() calls
I20260929 10:30:48.018802 1166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:30:48.019680 1166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c: Bootstrap complete.
I20260929 10:30:48.025794 1166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0888c6d1da944f7eb019027bade5962c" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37361 } }
I20260929 10:30:48.026400 1166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0888c6d1da944f7eb019027bade5962c, State: Initialized, Role: FOLLOWER
I20260929 10:30:48.027030 1166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0888c6d1da944f7eb019027bade5962c" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37361 } }
I20260929 10:30:48.027205 1166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:30:48.027334 1166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:30:48.027444 1166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [term 1 FOLLOWER]: Advancing to term 2
I20260929 10:30:48.030267 1166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0888c6d1da944f7eb019027bade5962c" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37361 } }
I20260929 10:30:48.030665 1166 leader_election.cc:302] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0888c6d1da944f7eb019027bade5962c; no voters:
I20260929 10:30:48.031041 1166 leader_election.cc:288] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 10:30:48.031210 1170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [term 2 FOLLOWER]: Leader election won for term 2
I20260929 10:30:48.031591 1170 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [term 2 LEADER]: Becoming Leader. State: Replica: 0888c6d1da944f7eb019027bade5962c, State: Running, Role: LEADER
I20260929 10:30:48.031991 1170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0888c6d1da944f7eb019027bade5962c" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37361 } }
I20260929 10:30:48.032559 1166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:30:48.033509 1170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0888c6d1da944f7eb019027bade5962c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0888c6d1da944f7eb019027bade5962c" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37361 } } }
I20260929 10:30:48.033735 1170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [sys.catalog]: This master's current role is: LEADER
I20260929 10:30:48.034025 1170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0888c6d1da944f7eb019027bade5962c. Latest consensus state: current_term: 2 leader_uuid: "0888c6d1da944f7eb019027bade5962c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0888c6d1da944f7eb019027bade5962c" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37361 } } }
I20260929 10:30:48.034211 1170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c [sys.catalog]: This master's current role is: LEADER
I20260929 10:30:48.034512 1179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:30:48.037923 1179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:30:48.038702 1166 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 10:30:48.039052 1179 catalog_manager.cc:1259] Loaded cluster ID: 648181a861e44b76aef00def809ea2f7
I20260929 10:30:48.039347 1179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:30:48.042178 1179 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:30:48.043000 1179 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c: Loaded TSK: 0
I20260929 10:30:48.044406 1179 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 10:30:48.151401 1168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:30:48.152381 1168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:30:48.152702 1168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:30:48.160470 1168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:30:48.160657 1168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:30:48.160815 1168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:30:48.170415 1168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.92.20:39913/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:34643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=46581
--tserver_master_addrs=127.20.92.62:37361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:30:48.172201 1168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:30:48.174084 1168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:30:48.190124 1203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:48.190049 1204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:48.190244 1168 server_base.cc:1034] running on GCE node
W20260929 10:30:48.190825 1206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:48.191411 1168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:30:48.192296 1168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:30:48.195706 1168 hybrid_clock.cc:648] HybridClock initialized: now 1790677848195394 us; error 333 us; skew 500 ppm
Sep 29 10:30:48 dist-test-slave-9cj5 krb5kdc[32602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677848, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:30:48.199959 1168 init.cc:377] Logged in from keytab as kudu/127.20.92.1@KRBTEST.COM (short username kudu)
I20260929 10:30:48.201325 1168 webserver.cc:460] Webserver started at http://127.20.92.1:46581/ using document root <none> and password file <none>
I20260929 10:30:48.202514 1168 fs_manager.cc:359] Metadata directory not provided
I20260929 10:30:48.202694 1168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 10:30:48 dist-test-slave-9cj5 krb5kdc[32602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677848, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for HTTP/127.20.92.20@KRBTEST.COM
I20260929 10:30:48.240273 1168 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.003s sys 0.000s
I20260929 10:30:48.243726 1214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:48.245503 1168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 10:30:48.245646 1168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "243e4c62a44e454782fc97ee4831c3af"
format_stamp: "Formatted at 2026-09-29 10:30:46 on dist-test-slave-9cj5"
server_key: "258be2483e2a0860c3d66d7645493e83"
server_key_iv: "5cdee28a09079cf9f8cf2911637f68e2"
server_key_version: "kuduclusterkey@0"
I20260929 10:30:48.246466 1168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:30:48.261832 1168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:30:48.266134 1168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:30:48.266423 1168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:30:48.267181 1168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:30:48.268378 1168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:30:48.268493 1168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:48.268704 1168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:30:48.268788 1168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:48.283855 1168 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:34643
I20260929 10:30:48.284027 1327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:34643 every 8 connection(s)
I20260929 10:30:48.285120 1168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
I20260929 10:30:48.286399 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 1168
I20260929 10:30:48.286957 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.92.20:39913/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:39887
--local_ip_for_outbound_sockets=127.20.92.2
--tserver_master_addrs=127.20.92.62:37361
--webserver_port=33519
--webserver_interface=127.20.92.2
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.92.20:39913/kms with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 10:30:48 dist-test-slave-9cj5 krb5kdc[32602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677848, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:30:48.310035 1331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:48.288030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:37361 (local address 127.20.92.1:54585)
0929 10:30:48.289099 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:48.289170 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:48.290409 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:48.293984 (+ 3575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:48.293995 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:48.294393 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:30:48.295195 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:48.295222 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:48.297700 (+ 2478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:48.297705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:48.299168 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:48.299183 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:48.300348 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:48.300354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:48.300558 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:48.301368 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:30:48.301392 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:30:48.303274 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:30:48.306114 (+ 2840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:48.306121 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:48.306137 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:48.306495 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:48.306935 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:48.306939 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:48.306941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:48.307111 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:48.307949 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:30:48.307956 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:30:48.308295 (+ 339us) client_negotiation.cc:769] Sending connection context
0929 10:30:48.308491 (+ 196us) client_negotiation.cc:240] Negotiation successful
0929 10:30:48.308859 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"thread_start_us":149,"threads_started":1}
I20260929 10:30:48.312752 1332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:48.289727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:54585 (local address 127.20.92.62:37361)
0929 10:30:48.292426 (+ 2699us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:48.292450 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:48.292503 (+ 53us) server_negotiation.cc:407] Connection header received
0929 10:30:48.292863 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:48.292882 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:48.293375 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:48.295548 (+ 2173us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:48.295923 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:48.297494 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:48.299363 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:48.300131 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:48.302255 (+ 2124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:48.303545 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:48.303553 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:48.303573 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:48.303610 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:48.305962 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:48.306654 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:48.306660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:48.306677 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:48.306818 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:48.307252 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:48.307255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:48.307257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:48.307809 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:48.311158 (+ 3349us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:48.311362 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 10:30:48.311713 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2466,"thread_start_us":125,"threads_started":1}
I20260929 10:30:48.318638 1328 heartbeater.cc:347] Connected to a master server at 127.20.92.62:37361
I20260929 10:30:48.318939 1328 heartbeater.cc:464] Registering TS with master...
I20260929 10:30:48.319576 1328 heartbeater.cc:511] Master 127.20.92.62:37361 requested a full tablet report, sending...
W20260929 10:30:48.322346 1328 heartbeater.cc:644] Failed to heartbeat to 127.20.92.62:37361 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 10:30:48.322489 1328 heartbeater.cc:464] Registering TS with master...
I20260929 10:30:48.322714 1328 heartbeater.cc:511] Master 127.20.92.62:37361 requested a full tablet report, sending...
I20260929 10:30:48.323814 1328 heartbeater.cc:464] Registering TS with master...
I20260929 10:30:48.324002 1328 heartbeater.cc:511] Master 127.20.92.62:37361 requested a full tablet report, sending...
W20260929 10:30:48.324898 1328 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 10:30:48.468459 1330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:30:48.468905 1330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:30:48.469031 1330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:30:48.474798 1330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:30:48.474987 1330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:30:48.475459 1330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:30:48.483639 1330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.92.20:39913/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:39887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=33519
--tserver_master_addrs=127.20.92.62:37361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:30:48.485404 1330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:30:48.487080 1330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:30:48.501088 1338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:48.501235 1337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:48.501088 1340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:48.502784 1330 server_base.cc:1034] running on GCE node
I20260929 10:30:48.503383 1330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:30:48.504317 1330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:30:48.505537 1330 hybrid_clock.cc:648] HybridClock initialized: now 1790677848505500 us; error 55 us; skew 500 ppm
Sep 29 10:30:48 dist-test-slave-9cj5 krb5kdc[32602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677848, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:30:48.509442 1330 init.cc:377] Logged in from keytab as kudu/127.20.92.2@KRBTEST.COM (short username kudu)
I20260929 10:30:48.510850 1330 webserver.cc:460] Webserver started at http://127.20.92.2:33519/ using document root <none> and password file <none>
I20260929 10:30:48.511528 1330 fs_manager.cc:359] Metadata directory not provided
I20260929 10:30:48.511615 1330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 10:30:48 dist-test-slave-9cj5 krb5kdc[32602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677848, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for HTTP/127.20.92.20@KRBTEST.COM
I20260929 10:30:48.539505 1330 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.003s sys 0.000s
I20260929 10:30:48.542833 1348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:48.544514 1330 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 10:30:48.544691 1330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "f7e3a29a14394eb0a11aaf06d4cec6e4"
format_stamp: "Formatted at 2026-09-29 10:30:47 on dist-test-slave-9cj5"
server_key: "9227ade540e6fb8f765c0667e6c10bd4"
server_key_iv: "8bd348aae9d848bf9d5815c98ec8bf74"
server_key_version: "kuduclusterkey@0"
I20260929 10:30:48.545434 1330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:30:48.565691 1330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:30:48.569406 1330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:30:48.569656 1330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:30:48.570636 1330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:30:48.571714 1330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:30:48.571791 1330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:48.571879 1330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:30:48.571939 1330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:48.586011 1330 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:39887
I20260929 10:30:48.586813 1461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:39887 every 8 connection(s)
I20260929 10:30:48.587598 1330 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
I20260929 10:30:48.589118 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 1330
I20260929 10:30:48.589633 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.92.20:39913/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:41725
--local_ip_for_outbound_sockets=127.20.92.3
--tserver_master_addrs=127.20.92.62:37361
--webserver_port=43735
--webserver_interface=127.20.92.3
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.92.20:39913/kms with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 10:30:48 dist-test-slave-9cj5 krb5kdc[32602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677848, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:30:48.609045 1332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:48.589972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:39017 (local address 127.20.92.62:37361)
0929 10:30:48.590147 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:48.590153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:48.593560 (+ 3407us) server_negotiation.cc:407] Connection header received
0929 10:30:48.594957 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:48.594961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:48.595051 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:48.595131 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:48.597669 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:48.598542 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:48.600431 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:48.600859 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:48.600951 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:48.604313 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:48.604317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:48.604320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:48.604358 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:48.606143 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:48.606881 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:48.606885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:48.606887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:48.606988 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:48.607489 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:48.607493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:48.607495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:48.607765 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:48.607846 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:48.608713 (+ 867us) server_negotiation.cc:299] Negotiation successful
0929 10:30:48.608865 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 10:30:48.611323 1465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:48.591193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:37361 (local address 127.20.92.2:39017)
0929 10:30:48.593346 (+ 2153us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:48.593417 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:48.594660 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:48.595618 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:48.595631 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:48.596022 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:30:48.597204 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:48.597233 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:48.598750 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:48.598753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:48.600250 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:48.600264 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:48.601198 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:48.601202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:48.601430 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:48.602239 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:30:48.602263 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:30:48.604045 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:30:48.606301 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:48.606309 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:48.606331 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:48.606729 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:48.607111 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:48.607114 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:48.607116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:48.607364 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:48.608008 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:30:48.608016 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:30:48.608408 (+ 392us) client_negotiation.cc:769] Sending connection context
0929 10:30:48.609772 (+ 1364us) client_negotiation.cc:240] Negotiation successful
0929 10:30:48.610206 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1878,"thread_start_us":172,"threads_started":1}
I20260929 10:30:48.613034 1462 heartbeater.cc:347] Connected to a master server at 127.20.92.62:37361
I20260929 10:30:48.613307 1462 heartbeater.cc:464] Registering TS with master...
I20260929 10:30:48.614043 1462 heartbeater.cc:511] Master 127.20.92.62:37361 requested a full tablet report, sending...
W20260929 10:30:48.615996 1462 heartbeater.cc:644] Failed to heartbeat to 127.20.92.62:37361 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 10:30:48.616124 1462 heartbeater.cc:464] Registering TS with master...
I20260929 10:30:48.616322 1462 heartbeater.cc:511] Master 127.20.92.62:37361 requested a full tablet report, sending...
I20260929 10:30:48.617241 1462 heartbeater.cc:464] Registering TS with master...
I20260929 10:30:48.617424 1462 heartbeater.cc:511] Master 127.20.92.62:37361 requested a full tablet report, sending...
W20260929 10:30:48.618371 1462 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 10:30:48.768235 1464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:30:48.768749 1464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:30:48.768868 1464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:30:48.775513 1464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:30:48.775691 1464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:30:48.775835 1464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:30:48.784977 1464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.92.20:39913/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:41725
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=43735
--tserver_master_addrs=127.20.92.62:37361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:30:48.786516 1464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:30:48.787985 1464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:30:48.801183 1474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:48.801383 1471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:48.801177 1472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:48.805876 1464 server_base.cc:1034] running on GCE node
I20260929 10:30:48.806515 1464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:30:48.807404 1464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:30:48.810710 1464 hybrid_clock.cc:648] HybridClock initialized: now 1790677848810634 us; error 104 us; skew 500 ppm
Sep 29 10:30:48 dist-test-slave-9cj5 krb5kdc[32602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677848, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:30:48.813824 1464 init.cc:377] Logged in from keytab as kudu/127.20.92.3@KRBTEST.COM (short username kudu)
I20260929 10:30:48.814802 1464 webserver.cc:460] Webserver started at http://127.20.92.3:43735/ using document root <none> and password file <none>
I20260929 10:30:48.815493 1464 fs_manager.cc:359] Metadata directory not provided
I20260929 10:30:48.815601 1464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 10:30:48 dist-test-slave-9cj5 krb5kdc[32602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677848, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for HTTP/127.20.92.20@KRBTEST.COM
I20260929 10:30:48.843101 1464 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 10:30:48.847579 1482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:48.849632 1464 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 10:30:48.849858 1464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "ae32ad291ff14447a2e7737392679b18"
format_stamp: "Formatted at 2026-09-29 10:30:47 on dist-test-slave-9cj5"
server_key: "1d478818a5fa16cd1dcd00d3c77cd11b"
server_key_iv: "e0340415fe3c7350253e4e270b1b5c03"
server_key_version: "kuduclusterkey@0"
I20260929 10:30:48.850708 1464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:30:48.864727 1464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:30:48.868839 1464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:30:48.869138 1464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:30:48.870028 1464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:30:48.871253 1464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:30:48.871369 1464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:48.871474 1464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:30:48.871546 1464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:48.885407 1464 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:41725
I20260929 10:30:48.885478 1595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:41725 every 8 connection(s)
I20260929 10:30:48.886618 1464 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:30:48.896957 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 1464
Sep 29 10:30:48 dist-test-slave-9cj5 krb5kdc[32602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677848, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:30:48.905434 1332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:48.889036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:33675 (local address 127.20.92.62:37361)
0929 10:30:48.889220 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:48.889226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:48.890358 (+ 1132us) server_negotiation.cc:407] Connection header received
0929 10:30:48.891637 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:48.891641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:48.891703 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:48.891813 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:48.893612 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:48.894320 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:48.896237 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:48.896483 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:48.896582 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:48.900399 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:48.900403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:48.900406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:48.900433 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:48.902117 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:48.902915 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:48.902922 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:48.902927 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:48.903016 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:48.903410 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:48.903417 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:48.903421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:48.903648 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:48.903755 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:48.904873 (+ 1118us) server_negotiation.cc:299] Negotiation successful
0929 10:30:48.905268 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 10:30:48.908193 1598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:48.889461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:37361 (local address 127.20.92.3:33675)
0929 10:30:48.890153 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:48.890230 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:48.891379 (+ 1149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:48.892020 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:48.892030 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:48.892434 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:30:48.893309 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:48.893336 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:48.894488 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:48.894492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:48.895963 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:48.895977 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:48.896693 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:48.896698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:48.896915 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:48.897938 (+ 1023us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:30:48.897960 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:30:48.900193 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:30:48.902250 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:48.902256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:48.902272 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:48.902784 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:48.903127 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:48.903131 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:48.903133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:48.903306 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:48.903767 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:30:48.903776 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:30:48.904519 (+ 743us) client_negotiation.cc:769] Sending connection context
0929 10:30:48.904806 (+ 287us) client_negotiation.cc:240] Negotiation successful
0929 10:30:48.905199 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":168,"threads_started":1}
I20260929 10:30:48.909554 1596 heartbeater.cc:347] Connected to a master server at 127.20.92.62:37361
I20260929 10:30:48.909843 1596 heartbeater.cc:464] Registering TS with master...
I20260929 10:30:48.910362 1596 heartbeater.cc:511] Master 127.20.92.62:37361 requested a full tablet report, sending...
I20260929 10:30:48.911358 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:48.898175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58418 (local address 127.20.92.62:37361)
0929 10:30:48.898855 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:48.898863 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:48.898874 (+ 11us) server_negotiation.cc:407] Connection header received
0929 10:30:48.898906 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:48.898908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:48.898947 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:48.899068 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:48.899874 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:48.900669 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:48.901936 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:48.902321 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:48.902423 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:48.904074 (+ 1651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:48.904081 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:48.904085 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:48.904112 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:48.906019 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:48.906558 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:48.906562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:48.906564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:48.906658 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:48.906998 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:48.907002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:48.907004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:48.907273 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:48.910911 (+ 3638us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:48.911012 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 10:30:48.911168 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":95,"threads_started":1}
W20260929 10:30:48.912524 1596 heartbeater.cc:644] Failed to heartbeat to 127.20.92.62:37361 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 10:30:48.912752 1596 heartbeater.cc:464] Registering TS with master...
I20260929 10:30:48.912961 1596 heartbeater.cc:511] Master 127.20.92.62:37361 requested a full tablet report, sending...
I20260929 10:30:48.914024 1596 heartbeater.cc:464] Registering TS with master...
I20260929 10:30:48.914217 1596 heartbeater.cc:511] Master 127.20.92.62:37361 requested a full tablet report, sending...
W20260929 10:30:48.915050 1596 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 10:30:49.325176 1328 heartbeater.cc:464] Registering TS with master...
I20260929 10:30:49.325533 1328 heartbeater.cc:511] Master 127.20.92.62:37361 requested a full tablet report, sending...
I20260929 10:30:49.326917 1130 ts_manager.cc:194] Registered new tserver with Master: 243e4c62a44e454782fc97ee4831c3af (127.20.92.1:34643)
I20260929 10:30:49.328619 1130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.1@KRBTEST.COM'} at 127.20.92.1:54585
Sep 29 10:30:49 dist-test-slave-9cj5 krb5kdc[32602](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790677849, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:30:49.618604 1462 heartbeater.cc:464] Registering TS with master...
I20260929 10:30:49.618918 1462 heartbeater.cc:511] Master 127.20.92.62:37361 requested a full tablet report, sending...
I20260929 10:30:49.619748 1130 ts_manager.cc:194] Registered new tserver with Master: f7e3a29a14394eb0a11aaf06d4cec6e4 (127.20.92.2:39887)
I20260929 10:30:49.620580 1130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.2@KRBTEST.COM'} at 127.20.92.2:39017
Sep 29 10:30:49 dist-test-slave-9cj5 krb5kdc[32602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677849, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for HTTP/127.20.92.20@KRBTEST.COM
I20260929 10:30:49.915325 1596 heartbeater.cc:464] Registering TS with master...
I20260929 10:30:49.915732 1596 heartbeater.cc:511] Master 127.20.92.62:37361 requested a full tablet report, sending...
I20260929 10:30:49.916796 1130 ts_manager.cc:194] Registered new tserver with Master: ae32ad291ff14447a2e7737392679b18 (127.20.92.3:41725)
I20260929 10:30:49.917593 1130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.3@KRBTEST.COM'} at 127.20.92.3:33675
I20260929 10:30:49.921890 20848 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:30:49.933693 1618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:49.923569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58426 (local address 127.20.92.62:37361)
0929 10:30:49.923821 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:49.923826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:49.923952 (+ 126us) server_negotiation.cc:407] Connection header received
0929 10:30:49.924058 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:49.924061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:49.924107 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:49.924210 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:49.925083 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:49.925957 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:49.927131 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:49.927512 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:49.927616 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:49.929371 (+ 1755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:49.929375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:49.929378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:49.929406 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:49.931342 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:49.931865 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:49.931869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:49.931871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:49.931958 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:49.932254 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:49.932257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:49.932259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:49.932506 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:49.932630 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:49.933331 (+ 701us) server_negotiation.cc:299] Negotiation successful
0929 10:30:49.933462 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":97,"threads_started":1}
I20260929 10:30:49.938263 1130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:30:49.943225 1130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:30:49.965516 1626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:49.955524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:34643 (local address 127.0.0.1:36210)
0929 10:30:49.957322 (+ 1798us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:49.957335 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:49.957445 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:49.958171 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:49.958177 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:49.958216 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:30:49.958374 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:49.958381 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:49.961475 (+ 3094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:49.961479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:49.963557 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:49.963569 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:49.964881 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:49.964885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:49.965018 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:49.965058 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 10:30:49.965182 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 10:30:49.965335 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1736,"mutex_wait_us":3,"spinlock_wait_cycles":10752,"thread_start_us":86,"threads_started":1}
I20260929 10:30:49.966125 1627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:49.955946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36210 (local address 127.20.92.1:34643)
0929 10:30:49.956969 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:49.956976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:49.957450 (+ 474us) server_negotiation.cc:407] Connection header received
0929 10:30:49.957552 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:49.957557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:49.957825 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:49.957975 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:30:49.958616 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:49.961340 (+ 2724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:49.963707 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:49.964557 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:49.964662 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:49.964818 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:49.965890 (+ 1072us) server_negotiation.cc:299] Negotiation successful
0929 10:30:49.965999 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":912,"thread_start_us":106,"threads_started":1}
I20260929 10:30:49.967203 1628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:49.955992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40236 (local address 127.20.92.3:41725)
0929 10:30:49.957183 (+ 1191us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:49.957191 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:49.958829 (+ 1638us) server_negotiation.cc:407] Connection header received
0929 10:30:49.958909 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:49.958914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:49.959111 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:49.959258 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:30:49.959693 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:49.960824 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:49.963654 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:49.964548 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:49.964663 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:49.964822 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:49.966935 (+ 2113us) server_negotiation.cc:299] Negotiation successful
0929 10:30:49.967046 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1093,"thread_start_us":73,"threads_started":1}
I20260929 10:30:49.965821 1623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:49.955858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:41725 (local address 127.0.0.1:40236)
0929 10:30:49.958623 (+ 2765us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:49.958638 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:49.958727 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:49.959429 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:49.959432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:49.959450 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:30:49.959553 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:49.959561 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:49.960964 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:49.960968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:49.963505 (+ 2537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:49.963518 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:49.964895 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:49.964899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:49.965453 (+ 554us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:49.965480 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 10:30:49.965566 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 10:30:49.965647 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2704,"mutex_wait_us":1344,"thread_start_us":101,"threads_started":1}
I20260929 10:30:49.969816 1625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:49.955078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:39887 (local address 127.0.0.1:58514)
0929 10:30:49.956489 (+ 1411us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:49.956545 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:49.956683 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:49.958176 (+ 1493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:49.958179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:49.958219 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:30:49.958509 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:49.958514 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:49.965038 (+ 6524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:49.965041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:49.966783 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:49.966796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:49.969381 (+ 2585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:49.969385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:49.969484 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:49.969511 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 10:30:49.969562 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 10:30:49.969637 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1230,"mutex_wait_us":166,"spinlock_wait_cycles":4096,"thread_start_us":120,"threads_started":1}
I20260929 10:30:49.970786 1624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:49.955194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58514 (local address 127.20.92.2:39887)
0929 10:30:49.957686 (+ 2492us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:49.957692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:49.957717 (+ 25us) server_negotiation.cc:407] Connection header received
0929 10:30:49.957817 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:49.957822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:49.958004 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:49.958376 (+ 372us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:30:49.960574 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:49.963266 (+ 2692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:49.967485 (+ 4219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:49.969223 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:49.970311 (+ 1088us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:49.970476 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:49.970533 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 10:30:49.970637 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2388,"thread_start_us":109,"threads_started":1}
I20260929 10:30:49.970899 1529 tablet_service.cc:1467] Processing CreateTablet for tablet 5640cdaf329d43a9a2d3cc89a61dca63 (DEFAULT_TABLE table=test-table [id=8abc0cf60cba4fcbbabd419afd100856]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:30:49.972141 1529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5640cdaf329d43a9a2d3cc89a61dca63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:30:49.973335 1262 tablet_service.cc:1467] Processing CreateTablet for tablet 5640cdaf329d43a9a2d3cc89a61dca63 (DEFAULT_TABLE table=test-table [id=8abc0cf60cba4fcbbabd419afd100856]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:30:49.973335 1396 tablet_service.cc:1467] Processing CreateTablet for tablet 5640cdaf329d43a9a2d3cc89a61dca63 (DEFAULT_TABLE table=test-table [id=8abc0cf60cba4fcbbabd419afd100856]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:30:49.974406 1262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5640cdaf329d43a9a2d3cc89a61dca63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:30:49.974493 1396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5640cdaf329d43a9a2d3cc89a61dca63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:30:49.981688 1629 tablet_bootstrap.cc:492] T 5640cdaf329d43a9a2d3cc89a61dca63 P ae32ad291ff14447a2e7737392679b18: Bootstrap starting.
I20260929 10:30:49.984091 1630 tablet_bootstrap.cc:492] T 5640cdaf329d43a9a2d3cc89a61dca63 P 243e4c62a44e454782fc97ee4831c3af: Bootstrap starting.
I20260929 10:30:49.984959 1629 tablet_bootstrap.cc:654] T 5640cdaf329d43a9a2d3cc89a61dca63 P ae32ad291ff14447a2e7737392679b18: Neither blocks nor log segments found. Creating new log.
I20260929 10:30:49.985860 1629 log.cc:824] T 5640cdaf329d43a9a2d3cc89a61dca63 P ae32ad291ff14447a2e7737392679b18: Log is configured to *not* fsync() on all Append() calls
I20260929 10:30:49.986414 1630 tablet_bootstrap.cc:654] T 5640cdaf329d43a9a2d3cc89a61dca63 P 243e4c62a44e454782fc97ee4831c3af: Neither blocks nor log segments found. Creating new log.
I20260929 10:30:49.987252 1630 log.cc:824] T 5640cdaf329d43a9a2d3cc89a61dca63 P 243e4c62a44e454782fc97ee4831c3af: Log is configured to *not* fsync() on all Append() calls
I20260929 10:30:49.988036 1631 tablet_bootstrap.cc:492] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4: Bootstrap starting.
I20260929 10:30:49.989850 1630 tablet_bootstrap.cc:492] T 5640cdaf329d43a9a2d3cc89a61dca63 P 243e4c62a44e454782fc97ee4831c3af: No bootstrap required, opened a new log
I20260929 10:30:49.990114 1630 ts_tablet_manager.cc:1397] T 5640cdaf329d43a9a2d3cc89a61dca63 P 243e4c62a44e454782fc97ee4831c3af: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20260929 10:30:49.990440 1631 tablet_bootstrap.cc:654] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4: Neither blocks nor log segments found. Creating new log.
I20260929 10:30:49.991338 1631 log.cc:824] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4: Log is configured to *not* fsync() on all Append() calls
I20260929 10:30:49.994222 1631 tablet_bootstrap.cc:492] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4: No bootstrap required, opened a new log
I20260929 10:30:49.994448 1631 ts_tablet_manager.cc:1397] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 10:30:49.995213 1630 raft_consensus.cc:348] T 5640cdaf329d43a9a2d3cc89a61dca63 P 243e4c62a44e454782fc97ee4831c3af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7e3a29a14394eb0a11aaf06d4cec6e4" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39887 } } peers { permanent_uuid: "ae32ad291ff14447a2e7737392679b18" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 41725 } } peers { permanent_uuid: "243e4c62a44e454782fc97ee4831c3af" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 34643 } }
I20260929 10:30:49.995486 1630 raft_consensus.cc:374] T 5640cdaf329d43a9a2d3cc89a61dca63 P 243e4c62a44e454782fc97ee4831c3af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:30:49.995532 1630 raft_consensus.cc:729] T 5640cdaf329d43a9a2d3cc89a61dca63 P 243e4c62a44e454782fc97ee4831c3af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 243e4c62a44e454782fc97ee4831c3af, State: Initialized, Role: FOLLOWER
I20260929 10:30:49.996063 1630 consensus_queue.cc:260] T 5640cdaf329d43a9a2d3cc89a61dca63 P 243e4c62a44e454782fc97ee4831c3af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7e3a29a14394eb0a11aaf06d4cec6e4" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39887 } } peers { permanent_uuid: "ae32ad291ff14447a2e7737392679b18" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 41725 } } peers { permanent_uuid: "243e4c62a44e454782fc97ee4831c3af" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 34643 } }
I20260929 10:30:49.997188 1630 ts_tablet_manager.cc:1428] T 5640cdaf329d43a9a2d3cc89a61dca63 P 243e4c62a44e454782fc97ee4831c3af: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260929 10:30:49.997576 1328 heartbeater.cc:503] Master 127.20.92.62:37361 was elected leader, sending a full tablet report...
I20260929 10:30:49.999045 1631 raft_consensus.cc:348] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7e3a29a14394eb0a11aaf06d4cec6e4" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39887 } } peers { permanent_uuid: "ae32ad291ff14447a2e7737392679b18" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 41725 } } peers { permanent_uuid: "243e4c62a44e454782fc97ee4831c3af" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 34643 } }
I20260929 10:30:49.999457 1631 raft_consensus.cc:374] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:30:49.999565 1631 raft_consensus.cc:729] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7e3a29a14394eb0a11aaf06d4cec6e4, State: Initialized, Role: FOLLOWER
I20260929 10:30:50.000152 1631 consensus_queue.cc:260] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7e3a29a14394eb0a11aaf06d4cec6e4" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39887 } } peers { permanent_uuid: "ae32ad291ff14447a2e7737392679b18" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 41725 } } peers { permanent_uuid: "243e4c62a44e454782fc97ee4831c3af" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 34643 } }
I20260929 10:30:50.001422 1631 ts_tablet_manager.cc:1428] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 10:30:50.002194 1462 heartbeater.cc:503] Master 127.20.92.62:37361 was elected leader, sending a full tablet report...
I20260929 10:30:50.004598 1629 tablet_bootstrap.cc:492] T 5640cdaf329d43a9a2d3cc89a61dca63 P ae32ad291ff14447a2e7737392679b18: No bootstrap required, opened a new log
I20260929 10:30:50.004832 1629 ts_tablet_manager.cc:1397] T 5640cdaf329d43a9a2d3cc89a61dca63 P ae32ad291ff14447a2e7737392679b18: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.000s
I20260929 10:30:50.010277 1629 raft_consensus.cc:348] T 5640cdaf329d43a9a2d3cc89a61dca63 P ae32ad291ff14447a2e7737392679b18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7e3a29a14394eb0a11aaf06d4cec6e4" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39887 } } peers { permanent_uuid: "ae32ad291ff14447a2e7737392679b18" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 41725 } } peers { permanent_uuid: "243e4c62a44e454782fc97ee4831c3af" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 34643 } }
I20260929 10:30:50.010610 1629 raft_consensus.cc:374] T 5640cdaf329d43a9a2d3cc89a61dca63 P ae32ad291ff14447a2e7737392679b18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:30:50.010654 1629 raft_consensus.cc:729] T 5640cdaf329d43a9a2d3cc89a61dca63 P ae32ad291ff14447a2e7737392679b18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae32ad291ff14447a2e7737392679b18, State: Initialized, Role: FOLLOWER
I20260929 10:30:50.011209 1629 consensus_queue.cc:260] T 5640cdaf329d43a9a2d3cc89a61dca63 P ae32ad291ff14447a2e7737392679b18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7e3a29a14394eb0a11aaf06d4cec6e4" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39887 } } peers { permanent_uuid: "ae32ad291ff14447a2e7737392679b18" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 41725 } } peers { permanent_uuid: "243e4c62a44e454782fc97ee4831c3af" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 34643 } }
I20260929 10:30:50.012367 1629 ts_tablet_manager.cc:1428] T 5640cdaf329d43a9a2d3cc89a61dca63 P ae32ad291ff14447a2e7737392679b18: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 10:30:50.012732 1596 heartbeater.cc:503] Master 127.20.92.62:37361 was elected leader, sending a full tablet report...
W20260929 10:30:50.037880 1329 tablet.cc:2369] T 5640cdaf329d43a9a2d3cc89a61dca63 P 243e4c62a44e454782fc97ee4831c3af: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 10:30:50 dist-test-slave-9cj5 krb5kdc[32602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677849, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for HTTP/127.20.92.20@KRBTEST.COM
W20260929 10:30:50.090547 1463 tablet.cc:2369] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:30:50.138437 1597 tablet.cc:2369] T 5640cdaf329d43a9a2d3cc89a61dca63 P ae32ad291ff14447a2e7737392679b18: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:30:50.197904 1636 raft_consensus.cc:482] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:30:50.198160 1636 raft_consensus.cc:504] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7e3a29a14394eb0a11aaf06d4cec6e4" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39887 } } peers { permanent_uuid: "ae32ad291ff14447a2e7737392679b18" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 41725 } } peers { permanent_uuid: "243e4c62a44e454782fc97ee4831c3af" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 34643 } }
I20260929 10:30:50.199640 1636 leader_election.cc:288] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae32ad291ff14447a2e7737392679b18 (127.20.92.3:41725), 243e4c62a44e454782fc97ee4831c3af (127.20.92.1:34643)
I20260929 10:30:50.204941 1628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:50.200213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:34883 (local address 127.20.92.3:41725)
0929 10:30:50.200698 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:50.200701 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:50.200713 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:30:50.200756 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:50.200758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:50.200797 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:50.200895 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:30:50.201220 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:50.202019 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:50.203593 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:50.204191 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:50.204280 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:50.204353 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:50.204720 (+ 367us) server_negotiation.cc:299] Negotiation successful
0929 10:30:50.204805 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394}
I20260929 10:30:50.205732 1640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:50.199811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:41725 (local address 127.20.92.2:34883)
0929 10:30:50.200400 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:50.200417 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:50.200522 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:50.200911 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:50.200914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:50.200933 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:30:50.201082 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:50.201090 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:50.202177 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:50.202180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:50.203431 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:50.203445 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:50.204456 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:50.204459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:50.204559 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:50.204582 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 10:30:50.205466 (+ 884us) client_negotiation.cc:240] Negotiation successful
0929 10:30:50.205564 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":127,"threads_started":1}
I20260929 10:30:50.206622 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5640cdaf329d43a9a2d3cc89a61dca63" candidate_uuid: "f7e3a29a14394eb0a11aaf06d4cec6e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae32ad291ff14447a2e7737392679b18" is_pre_election: true
I20260929 10:30:50.206981 1550 raft_consensus.cc:2454] T 5640cdaf329d43a9a2d3cc89a61dca63 P ae32ad291ff14447a2e7737392679b18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7e3a29a14394eb0a11aaf06d4cec6e4 in term 0.
I20260929 10:30:50.207700 1349 leader_election.cc:302] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae32ad291ff14447a2e7737392679b18, f7e3a29a14394eb0a11aaf06d4cec6e4; no voters:
I20260929 10:30:50.208096 1636 raft_consensus.cc:2781] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:30:50.208173 1636 raft_consensus.cc:482] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:30:50.208212 1636 raft_consensus.cc:3037] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:30:50.209633 1636 raft_consensus.cc:504] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7e3a29a14394eb0a11aaf06d4cec6e4" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39887 } } peers { permanent_uuid: "ae32ad291ff14447a2e7737392679b18" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 41725 } } peers { permanent_uuid: "243e4c62a44e454782fc97ee4831c3af" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 34643 } }
I20260929 10:30:50.210105 1636 leader_election.cc:288] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4 [CANDIDATE]: Term 1 election: Requested vote from peers ae32ad291ff14447a2e7737392679b18 (127.20.92.3:41725), 243e4c62a44e454782fc97ee4831c3af (127.20.92.1:34643)
I20260929 10:30:50.210886 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5640cdaf329d43a9a2d3cc89a61dca63" candidate_uuid: "f7e3a29a14394eb0a11aaf06d4cec6e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae32ad291ff14447a2e7737392679b18"
I20260929 10:30:50.211030 1550 raft_consensus.cc:3037] T 5640cdaf329d43a9a2d3cc89a61dca63 P ae32ad291ff14447a2e7737392679b18 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:30:50.212435 1550 raft_consensus.cc:2454] T 5640cdaf329d43a9a2d3cc89a61dca63 P ae32ad291ff14447a2e7737392679b18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7e3a29a14394eb0a11aaf06d4cec6e4 in term 1.
I20260929 10:30:50.212949 1349 leader_election.cc:302] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae32ad291ff14447a2e7737392679b18, f7e3a29a14394eb0a11aaf06d4cec6e4; no voters:
I20260929 10:30:50.213212 1636 raft_consensus.cc:2781] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:30:50.213441 1636 raft_consensus.cc:686] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4 [term 1 LEADER]: Becoming Leader. State: Replica: f7e3a29a14394eb0a11aaf06d4cec6e4, State: Running, Role: LEADER
I20260929 10:30:50.213874 1636 consensus_queue.cc:237] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7e3a29a14394eb0a11aaf06d4cec6e4" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39887 } } peers { permanent_uuid: "ae32ad291ff14447a2e7737392679b18" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 41725 } } peers { permanent_uuid: "243e4c62a44e454782fc97ee4831c3af" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 34643 } }
I20260929 10:30:50.217605 1627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:50.201077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:34731 (local address 127.20.92.1:34643)
0929 10:30:50.201236 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:50.201241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:50.210470 (+ 9229us) server_negotiation.cc:407] Connection header received
0929 10:30:50.210607 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:50.210610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:50.210687 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:50.210780 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:30:50.213892 (+ 3112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:50.214657 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:50.216248 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:50.216867 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:50.216958 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:50.217030 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:50.217383 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 10:30:50.217467 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 10:30:50.218271 1641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:50.200229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:34643 (local address 127.20.92.2:34731)
0929 10:30:50.210349 (+ 10120us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:50.210367 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:50.210503 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:50.210925 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:50.210928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:50.210947 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:30:50.211078 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:50.211084 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:50.214841 (+ 3757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:50.214845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:50.216074 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:50.216087 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:50.217131 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:50.217134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:50.217244 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:50.217266 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 10:30:50.218021 (+ 755us) client_negotiation.cc:240] Negotiation successful
0929 10:30:50.218111 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10028,"thread_start_us":76,"threads_started":1}
I20260929 10:30:50.217999 1130 catalog_manager.cc:5495] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7e3a29a14394eb0a11aaf06d4cec6e4 (127.20.92.2). New cstate: current_term: 1 leader_uuid: "f7e3a29a14394eb0a11aaf06d4cec6e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7e3a29a14394eb0a11aaf06d4cec6e4" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae32ad291ff14447a2e7737392679b18" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 41725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "243e4c62a44e454782fc97ee4831c3af" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 34643 } health_report { overall_health: UNKNOWN } } }
I20260929 10:30:50.219096 1267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5640cdaf329d43a9a2d3cc89a61dca63" candidate_uuid: "f7e3a29a14394eb0a11aaf06d4cec6e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "243e4c62a44e454782fc97ee4831c3af" is_pre_election: true
I20260929 10:30:50.219419 1267 raft_consensus.cc:2454] T 5640cdaf329d43a9a2d3cc89a61dca63 P 243e4c62a44e454782fc97ee4831c3af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7e3a29a14394eb0a11aaf06d4cec6e4 in term 0.
I20260929 10:30:50.219775 1280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5640cdaf329d43a9a2d3cc89a61dca63" candidate_uuid: "f7e3a29a14394eb0a11aaf06d4cec6e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "243e4c62a44e454782fc97ee4831c3af"
I20260929 10:30:50.219879 1280 raft_consensus.cc:3037] T 5640cdaf329d43a9a2d3cc89a61dca63 P 243e4c62a44e454782fc97ee4831c3af [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:30:50.221297 1280 raft_consensus.cc:2454] T 5640cdaf329d43a9a2d3cc89a61dca63 P 243e4c62a44e454782fc97ee4831c3af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7e3a29a14394eb0a11aaf06d4cec6e4 in term 1.
I20260929 10:30:50.272359 1624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:50.266134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58518 (local address 127.20.92.2:39887)
0929 10:30:50.266261 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:50.266268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:50.266286 (+ 18us) server_negotiation.cc:407] Connection header received
0929 10:30:50.266389 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:50.266391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:50.266456 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:50.266613 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:30:50.266905 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:50.267793 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:50.269202 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:50.269618 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:50.270099 (+ 481us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:50.270290 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:30:50.270950 (+ 660us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:30:50.271083 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:50.272064 (+ 981us) server_negotiation.cc:299] Negotiation successful
0929 10:30:50.272148 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 10:30:50.287885 1280 raft_consensus.cc:1261] T 5640cdaf329d43a9a2d3cc89a61dca63 P 243e4c62a44e454782fc97ee4831c3af [term 1 FOLLOWER]: Refusing update from remote peer f7e3a29a14394eb0a11aaf06d4cec6e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:30:50.289098 1642 consensus_queue.cc:1035] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "243e4c62a44e454782fc97ee4831c3af" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 34643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:30:50.298381 1550 raft_consensus.cc:1261] T 5640cdaf329d43a9a2d3cc89a61dca63 P ae32ad291ff14447a2e7737392679b18 [term 1 FOLLOWER]: Refusing update from remote peer f7e3a29a14394eb0a11aaf06d4cec6e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:30:50.299384 1642 consensus_queue.cc:1035] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae32ad291ff14447a2e7737392679b18" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 41725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:30:50.316823 1648 mvcc.cc:205] Tried to move back new op lower bound from 7334616474768072704 to 7334616474481045504. Current Snapshot: MvccSnapshot[applied={T|T < 7334616474768072704}]
I20260929 10:30:50.321411 1645 mvcc.cc:205] Tried to move back new op lower bound from 7334616474768072704 to 7334616474481045504. Current Snapshot: MvccSnapshot[applied={T|T < 7334616474768072704}]
I20260929 10:30:50.337863 1130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:30:50.338079 1130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:30:50.342406 1130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0888c6d1da944f7eb019027bade5962c: Sending DeleteTablet for 3 replicas of tablet 5640cdaf329d43a9a2d3cc89a61dca63
I20260929 10:30:50.343602 1262 tablet_service.cc:1514] Processing DeleteTablet for tablet 5640cdaf329d43a9a2d3cc89a61dca63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:30:50 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:36210
I20260929 10:30:50.344779 1396 tablet_service.cc:1514] Processing DeleteTablet for tablet 5640cdaf329d43a9a2d3cc89a61dca63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:30:50 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:58514
I20260929 10:30:50.345306 1529 tablet_service.cc:1514] Processing DeleteTablet for tablet 5640cdaf329d43a9a2d3cc89a61dca63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:30:50 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:40236
I20260929 10:30:50.346993 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 1168
I20260929 10:30:50.351042 1655 tablet_replica.cc:330] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4: stopping tablet replica
I20260929 10:30:50.351428 1656 tablet_replica.cc:330] T 5640cdaf329d43a9a2d3cc89a61dca63 P ae32ad291ff14447a2e7737392679b18: stopping tablet replica
I20260929 10:30:50.355122 1656 raft_consensus.cc:2229] T 5640cdaf329d43a9a2d3cc89a61dca63 P ae32ad291ff14447a2e7737392679b18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:30:50.355580 1655 raft_consensus.cc:2229] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:30:50.356262 1655 raft_consensus.cc:2258] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:30:50.356727 1656 raft_consensus.cc:2258] T 5640cdaf329d43a9a2d3cc89a61dca63 P ae32ad291ff14447a2e7737392679b18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:30:50.359431 1655 ts_tablet_manager.cc:1905] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:30:50.360442 1656 ts_tablet_manager.cc:1905] T 5640cdaf329d43a9a2d3cc89a61dca63 P ae32ad291ff14447a2e7737392679b18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:30:50.376168 1656 ts_tablet_manager.cc:1918] T 5640cdaf329d43a9a2d3cc89a61dca63 P ae32ad291ff14447a2e7737392679b18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:30:50.376735 1655 ts_tablet_manager.cc:1918] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260929 10:30:50.377921 1115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.92.1:34643 (error 108)
I20260929 10:30:50.379041 1655 log.cc:1196] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-1/wal/wals/5640cdaf329d43a9a2d3cc89a61dca63
I20260929 10:30:50.379660 1655 ts_tablet_manager.cc:1939] T 5640cdaf329d43a9a2d3cc89a61dca63 P f7e3a29a14394eb0a11aaf06d4cec6e4: Deleting consensus metadata
I20260929 10:30:50.379850 1656 log.cc:1196] T 5640cdaf329d43a9a2d3cc89a61dca63 P ae32ad291ff14447a2e7737392679b18: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677679287822-20848-0/minicluster-data/ts-2/wal/wals/5640cdaf329d43a9a2d3cc89a61dca63
I20260929 10:30:50.380127 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 1330
I20260929 10:30:50.382005 1656 ts_tablet_manager.cc:1939] T 5640cdaf329d43a9a2d3cc89a61dca63 P ae32ad291ff14447a2e7737392679b18: Deleting consensus metadata
I20260929 10:30:50.383760 1114 catalog_manager.cc:4841] TS ae32ad291ff14447a2e7737392679b18 (127.20.92.3:41725): tablet 5640cdaf329d43a9a2d3cc89a61dca63 (table test-table [id=8abc0cf60cba4fcbbabd419afd100856]) successfully deleted
I20260929 10:30:50.384955 1625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:50.384621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:34643 (local address 127.0.0.1:36212)
0929 10:30:50.384758 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:50.384839 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.1:34643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 10:30:50.385205 1115 catalog_manager.cc:4584] TS 243e4c62a44e454782fc97ee4831c3af (127.20.92.1:34643): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5640cdaf329d43a9a2d3cc89a61dca63: Network error: Client connection negotiation failed: client connection to 127.20.92.1:34643: connect: Connection refused (error 111)
W20260929 10:30:50.392344 1625 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:30:50.391562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:39887 (local address 127.0.0.1:58534)
0929 10:30:50.391768 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:50.391783 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:50.391885 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:50.392226 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.2:39887: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":125}
I20260929 10:30:50.393136 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 1464
I20260929 10:30:50.400799 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 1097
I20260929 10:30:50.410264 20848 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 10:30:50.746055 20848 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 10:30:50.746201 20848 mini_ranger.cc:66] Stopping Ranger...
I20260929 10:30:51.082978 20848 mini_ranger.cc:68] Stopped Ranger
2026-09-29 10:30:51.083 UTC [32632] LOG: received smart shutdown request
2026-09-29 10:30:51.088 UTC [32632] LOG: background worker "logical replication launcher" (PID 32641) exited with exit code 1
2026-09-29 10:30:51.088 UTC [32636] LOG: shutting down
2026-09-29 10:30:51.265 UTC [32632] LOG: database system is shut down
2026-09-29T10:30:51Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (55729 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:30:51 dist-test-slave-9cj5 krb5kdc[1666](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:30:51 dist-test-slave-9cj5 krb5kdc[1666](info): set up 2 sockets
Sep 29 10:30:51 dist-test-slave-9cj5 krb5kdc[1666](info): commencing operation
krb5kdc: starting...
W20260929 10:30:52.486519 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:30:52 dist-test-slave-9cj5 krb5kdc[1666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:30:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:30:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
I20260929 10:30:52.621724 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:37481
--webserver_interface=127.20.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:46029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:37481,127.20.92.61:40645,127.20.92.60:42711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:30:52.735129 1679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:30:52.735440 1679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:30:52.735481 1679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:30:52.739729 1679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:30:52.739805 1679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:30:52.739826 1679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:30:52.739845 1679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:30:52.739861 1679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:30:52.745417 1679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:46029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:37481,127.20.92.61:40645,127.20.92.60:42711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:37481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:30:52.746615 1679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:30:52.747623 1679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:30:52.756575 1685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:52.756680 1687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:52.756835 1679 server_base.cc:1034] running on GCE node
W20260929 10:30:52.756846 1684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:52.757634 1679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:30:52.758880 1679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:30:52.760115 1679 hybrid_clock.cc:648] HybridClock initialized: now 1790677852760095 us; error 32 us; skew 500 ppm
Sep 29 10:30:52 dist-test-slave-9cj5 krb5kdc[1666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677852, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:30:52.763391 1679 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:30:52.764465 1679 webserver.cc:460] Webserver started at http://127.20.92.62:38559/ using document root <none> and password file <none>
I20260929 10:30:52.765084 1679 fs_manager.cc:359] Metadata directory not provided
I20260929 10:30:52.765195 1679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:30:52.765491 1679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:30:52.768036 1679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "9867c91722cd4f8d8489ec186c912bfd"
format_stamp: "Formatted at 2026-09-29 10:30:52 on dist-test-slave-9cj5"
server_key: "b31eb4ac230a99f8affefe9ec3c24ede"
server_key_iv: "0d4279d69dce3d47e74f20cf99d81fe6"
server_key_version: "encryptionkey@0"
I20260929 10:30:52.768678 1679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "9867c91722cd4f8d8489ec186c912bfd"
format_stamp: "Formatted at 2026-09-29 10:30:52 on dist-test-slave-9cj5"
server_key: "b31eb4ac230a99f8affefe9ec3c24ede"
server_key_iv: "0d4279d69dce3d47e74f20cf99d81fe6"
server_key_version: "encryptionkey@0"
I20260929 10:30:52.772953 1679 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 10:30:52.775864 1694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:52.777177 1679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:30:52.777346 1679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "9867c91722cd4f8d8489ec186c912bfd"
format_stamp: "Formatted at 2026-09-29 10:30:52 on dist-test-slave-9cj5"
server_key: "b31eb4ac230a99f8affefe9ec3c24ede"
server_key_iv: "0d4279d69dce3d47e74f20cf99d81fe6"
server_key_version: "encryptionkey@0"
I20260929 10:30:52.777480 1679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:30:52.794956 1679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:30:52.798259 1679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:30:52.798507 1679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:30:52.805740 1679 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:37481
I20260929 10:30:52.805791 1746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:37481 every 8 connection(s)
I20260929 10:30:52.806734 1679 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:30:52.808522 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 1679
I20260929 10:30:52.808689 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:30:52.808974 20848 external_mini_cluster.cc:1428] Setting key 99349e860920b3d285d4d4b4e9e864f4
I20260929 10:30:52.809813 1747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:30:52.814035 1747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37481 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.92.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.61@KRBTEST.COM" created.
Sep 29 10:30:52 dist-test-slave-9cj5 krb5kdc[1666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677852, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:30:52.830476 1751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:52.816383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54762 (local address 127.20.92.62:37481)
0929 10:30:52.816905 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:52.816920 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:52.817203 (+ 283us) server_negotiation.cc:407] Connection header received
0929 10:30:52.818366 (+ 1163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:52.818371 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:52.818557 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:52.818718 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:52.819707 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:52.820631 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:52.822184 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:52.822560 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:52.822685 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:52.825421 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:52.825426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:52.825437 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:52.825457 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:52.827166 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:52.827676 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:52.827682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:52.827683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:52.827795 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:52.828108 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:52.828113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:52.828114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:52.828548 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:52.828689 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:52.829069 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 10:30:52.829439 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":348,"spinlock_wait_cycles":52480,"thread_start_us":154,"threads_started":1}
I20260929 10:30:52.830476 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:52.816383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:37481 (local address 127.0.0.1:54762)
0929 10:30:52.817019 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:52.817066 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:52.818079 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:52.818831 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:52.818836 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:52.819161 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:30:52.819531 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:52.819550 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:52.820762 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:52.820765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:52.822039 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:52.822054 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:52.822725 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:52.822737 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:52.822905 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:52.823505 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:30:52.823531 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:30:52.825253 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:30:52.827313 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:52.827324 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:52.827335 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:52.827559 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:52.827896 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:52.827899 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:52.827901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:52.828014 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:52.828677 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:30:52.828684 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:30:52.828824 (+ 140us) client_negotiation.cc:769] Sending connection context
0929 10:30:52.828998 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 10:30:52.829360 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":154,"threads_started":1}
I20260929 10:30:52.833244 1747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.92.61" port: 40645 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:30:52.834192 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:52.833822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:40645 (local address 127.0.0.1:47888)
0929 10:30:52.834001 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:52.834121 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.61:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 10:30:52.834432 1698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.92.61:40645: connect: Connection refused (error 111)
I20260929 10:30:52.835441 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:52.835195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:40645 (local address 127.0.0.1:47902)
0929 10:30:52.835331 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:52.835382 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.61:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 10:30:52.835583 1747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.92.61:40645: Network error: Client connection negotiation failed: client connection to 127.20.92.61:40645: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.61.keytab.
Entry for principal kudu/127.20.92.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.61.keytab.
I20260929 10:30:52.837139 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.61:40645
--webserver_interface=127.20.92.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.61.keytab
--principal=kudu/127.20.92.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:46029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:37481,127.20.92.61:40645,127.20.92.60:42711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:30:52.884874 1747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.92.61" port: 40645 } attempt: 1
I20260929 10:30:52.885794 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:52.885436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:40645 (local address 127.0.0.1:47918)
0929 10:30:52.885595 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:52.885669 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.61:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 10:30:52.886547 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:52.886332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:40645 (local address 127.0.0.1:47932)
0929 10:30:52.886437 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:52.886484 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.61:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 10:30:52.886726 1747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.92.61:40645: Network error: Client connection negotiation failed: client connection to 127.20.92.61:40645: connect: Connection refused (error 111)
W20260929 10:30:52.949874 1754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:30:52.950188 1754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:30:52.950230 1754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:30:52.954363 1754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:30:52.954428 1754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:30:52.954450 1754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:30:52.954468 1754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:30:52.954485 1754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:30:52.954923 1747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.92.61" port: 40645 } attempt: 2
I20260929 10:30:52.955772 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:52.955445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:40645 (local address 127.0.0.1:47944)
0929 10:30:52.955630 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:52.955697 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.61:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 10:30:52.956573 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:52.956357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:40645 (local address 127.0.0.1:47950)
0929 10:30:52.956466 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:52.956513 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.61:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 10:30:52.956808 1747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.92.61:40645: Network error: Client connection negotiation failed: client connection to 127.20.92.61:40645: connect: Connection refused (error 111)
I20260929 10:30:52.961058 1754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:46029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:37481,127.20.92.61:40645,127.20.92.60:42711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.61.keytab
--principal=kudu/127.20.92.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.61:40645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.92.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:30:52.962280 1754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:30:52.963333 1754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:30:52.972072 1759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:52.972069 1760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:52.972189 1754 server_base.cc:1034] running on GCE node
W20260929 10:30:52.972069 1762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:52.972887 1754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:30:52.974164 1754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:30:52.975358 1754 hybrid_clock.cc:648] HybridClock initialized: now 1790677852975342 us; error 27 us; skew 500 ppm
Sep 29 10:30:52 dist-test-slave-9cj5 krb5kdc[1666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677852, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:30:52.978248 1754 init.cc:377] Logged in from keytab as kudu/127.20.92.61@KRBTEST.COM (short username kudu)
I20260929 10:30:52.979189 1754 webserver.cc:460] Webserver started at http://127.20.92.61:39973/ using document root <none> and password file <none>
I20260929 10:30:52.979763 1754 fs_manager.cc:359] Metadata directory not provided
I20260929 10:30:52.979856 1754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:30:52.980089 1754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:30:52.982409 1754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-1/data/instance:
uuid: "0fc62a11dea9440895fe42e57967c29b"
format_stamp: "Formatted at 2026-09-29 10:30:52 on dist-test-slave-9cj5"
server_key: "940be7cfb3a50da43f0f58a0cd28ec40"
server_key_iv: "f03acaab05c1e1b885f7db795a3fce57"
server_key_version: "encryptionkey@0"
I20260929 10:30:52.982991 1754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-1/wal/instance:
uuid: "0fc62a11dea9440895fe42e57967c29b"
format_stamp: "Formatted at 2026-09-29 10:30:52 on dist-test-slave-9cj5"
server_key: "940be7cfb3a50da43f0f58a0cd28ec40"
server_key_iv: "f03acaab05c1e1b885f7db795a3fce57"
server_key_version: "encryptionkey@0"
I20260929 10:30:52.986829 1754 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 10:30:52.989418 1769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:52.990759 1754 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:30:52.990907 1754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-1/wal
uuid: "0fc62a11dea9440895fe42e57967c29b"
format_stamp: "Formatted at 2026-09-29 10:30:52 on dist-test-slave-9cj5"
server_key: "940be7cfb3a50da43f0f58a0cd28ec40"
server_key_iv: "f03acaab05c1e1b885f7db795a3fce57"
server_key_version: "encryptionkey@0"
I20260929 10:30:52.991029 1754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:30:53.003376 1754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:30:53.006901 1754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:30:53.007126 1754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:30:53.014946 1754 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.61:40645
I20260929 10:30:53.014971 1821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.61:40645 every 8 connection(s)
I20260929 10:30:53.015949 1754 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-1/data/info.pb
I20260929 10:30:53.018746 1822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:30:53.022514 1822 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37481 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:30:53.023777 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 1754
I20260929 10:30:53.023911 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-1/wal/instance
I20260929 10:30:53.024197 20848 external_mini_cluster.cc:1428] Setting key be21cde5998f278e1525728ae702c66a
WARNING: no policy specified for kudu/127.20.92.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.60@KRBTEST.COM" created.
Sep 29 10:30:53 dist-test-slave-9cj5 krb5kdc[1666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677852, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.61@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:30:53.040869 1751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.024566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54774 (local address 127.20.92.62:37481)
0929 10:30:53.024728 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:53.024732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:53.025777 (+ 1045us) server_negotiation.cc:407] Connection header received
0929 10:30:53.027184 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:53.027187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:53.027256 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:53.027333 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:53.028536 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.029076 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.030307 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.030554 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.030672 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:53.035847 (+ 5175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:53.035851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:53.035853 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:53.035882 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:53.038539 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:53.039198 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:53.039201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:53.039202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:53.039281 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:53.039665 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:53.039668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:53.039670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:53.039864 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:53.039980 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:53.040544 (+ 564us) server_negotiation.cc:299] Negotiation successful
0929 10:30:53.040688 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 10:30:53.041694 1824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.025032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:37481 (local address 127.0.0.1:54774)
0929 10:30:53.025561 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:53.025605 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:53.026905 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:53.027520 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:53.027527 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:53.027906 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:30:53.028376 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:53.028390 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.029235 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.029239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:53.030180 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:53.030190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.030719 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.030723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:53.030889 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:53.031517 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:30:53.031533 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:30:53.035623 (+ 4090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:30:53.038703 (+ 3080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:53.038710 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:53.038730 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:53.039069 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:53.039387 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:53.039390 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:53.039392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:53.039561 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:53.039981 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:30:53.039991 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:30:53.040284 (+ 293us) client_negotiation.cc:769] Sending connection context
0929 10:30:53.040479 (+ 195us) client_negotiation.cc:240] Negotiation successful
0929 10:30:53.040831 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":151,"threads_started":1}
I20260929 10:30:53.043294 1822 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.92.61" port: 40645 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:30:53.047987 1747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.92.61" port: 40645 } attempt: 3
Sep 29 10:30:53 dist-test-slave-9cj5 krb5kdc[1666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677852, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.61@KRBTEST.COM for kudu/127.20.92.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.60.keytab.
Entry for principal kudu/127.20.92.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.60.keytab.
I20260929 10:30:53.052870 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.60:42711
--webserver_interface=127.20.92.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.60.keytab
--principal=kudu/127.20.92.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:46029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:37481,127.20.92.61:40645,127.20.92.60:42711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 10:30:53 dist-test-slave-9cj5 krb5kdc[1666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677852, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for kudu/127.20.92.61@KRBTEST.COM
I20260929 10:30:53.054255 1824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.043828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:40645 (local address 127.0.0.1:47958)
0929 10:30:53.043927 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:53.043941 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:53.044018 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:53.044700 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:53.044703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:53.044855 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:30:53.044946 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:53.044951 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.046148 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.046153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:53.046727 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:53.046736 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.047436 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.047439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:53.047531 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:53.048075 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:30:53.048090 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:30:53.049576 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:30:53.051923 (+ 2347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:53.051926 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:53.051928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:53.052176 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:53.052949 (+ 773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:53.052953 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:53.052955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:53.053054 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:53.053663 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:30:53.053666 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:30:53.053820 (+ 154us) client_negotiation.cc:769] Sending connection context
0929 10:30:53.053938 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 10:30:53.054070 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 10:30:53.054549 1827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.043920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47958 (local address 127.20.92.61:40645)
0929 10:30:53.044306 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:53.044317 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:53.044336 (+ 19us) server_negotiation.cc:407] Connection header received
0929 10:30:53.044372 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:53.044376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:53.044524 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:53.044685 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:53.045086 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.045988 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.046885 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.047328 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.047410 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:53.049715 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:53.049728 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:53.049781 (+ 53us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:53.049813 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:53.051807 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:53.052701 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:53.052710 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:53.052712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:53.052808 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:53.053199 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:53.053202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:53.053203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:53.053534 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:53.053688 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:53.054026 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 10:30:53.054160 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":168,"threads_started":1}
I20260929 10:30:53.056435 1822 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.92.60" port: 42711 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:30:53.057152 1824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.056893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:42711 (local address 127.0.0.1:44244)
0929 10:30:53.057005 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:53.057085 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:42711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 10:30:53.057372 1771 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.92.60:42711: connect: Connection refused (error 111)
I20260929 10:30:53.058197 1824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.057992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:42711 (local address 127.0.0.1:44256)
0929 10:30:53.058094 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:53.058137 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:42711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 10:30:53.058415 1822 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.92.60:42711: Network error: Client connection negotiation failed: client connection to 127.20.92.60:42711: connect: Connection refused (error 111)
I20260929 10:30:53.059060 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.048485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:40645 (local address 127.0.0.1:47964)
0929 10:30:53.048597 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:53.048614 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:53.048723 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:53.049183 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:53.049187 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:53.049393 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:30:53.049556 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:53.049565 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.050843 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.050850 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:53.051613 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:53.051623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.052303 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.052309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:53.052439 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:53.052919 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:30:53.052941 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:30:53.054883 (+ 1942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:30:53.057185 (+ 2302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:53.057192 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:53.057197 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:53.057502 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:53.057910 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:53.057917 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:53.057922 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:53.058032 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:53.058643 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:30:53.058645 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:30:53.058742 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 10:30:53.058835 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 10:30:53.058947 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 10:30:53.059356 1828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.048614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47964 (local address 127.20.92.61:40645)
0929 10:30:53.048970 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:53.048976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:53.048994 (+ 18us) server_negotiation.cc:407] Connection header received
0929 10:30:53.049022 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:53.049024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:53.049056 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:53.049128 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:53.049697 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.050687 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.051784 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.052143 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.052279 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:53.055059 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:53.055063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:53.055066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:53.055100 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:53.057041 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:53.057657 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:53.057663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:53.057668 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:53.057792 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:53.058170 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:53.058177 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:53.058182 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:53.058471 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:53.058651 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:53.058934 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 10:30:53.059078 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":160,"threads_started":1}
I20260929 10:30:53.060107 1747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.92.60" port: 42711 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:30:53.060803 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.060577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:42711 (local address 127.0.0.1:44268)
0929 10:30:53.060685 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:53.060734 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:42711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 10:30:53.061403 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.061167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:42711 (local address 127.0.0.1:44274)
0929 10:30:53.061301 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:53.061345 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:42711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260929 10:30:53.061542 1747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.92.60:42711: Network error: Client connection negotiation failed: client connection to 127.20.92.60:42711: connect: Connection refused (error 111)
I20260929 10:30:53.092718 1747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.92.60" port: 42711 } attempt: 1
I20260929 10:30:53.093683 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.093347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:42711 (local address 127.0.0.1:44290)
0929 10:30:53.093530 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:53.093593 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:42711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 10:30:53.094561 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.094284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:42711 (local address 127.0.0.1:44300)
0929 10:30:53.094391 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:53.094436 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:42711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 10:30:53.094714 1747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.92.60:42711: Network error: Client connection negotiation failed: client connection to 127.20.92.60:42711: connect: Connection refused (error 111)
I20260929 10:30:53.107707 1822 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.92.60" port: 42711 } attempt: 1
I20260929 10:30:53.108599 1824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.108307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:42711 (local address 127.0.0.1:44306)
0929 10:30:53.108447 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:53.108521 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:42711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 10:30:53.109258 1824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.109060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:42711 (local address 127.0.0.1:44318)
0929 10:30:53.109160 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:53.109209 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:42711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 10:30:53.109380 1822 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.92.60:42711: Network error: Client connection negotiation failed: client connection to 127.20.92.60:42711: connect: Connection refused (error 111)
I20260929 10:30:53.169925 1747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.92.60" port: 42711 } attempt: 2
I20260929 10:30:53.170809 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.170460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:42711 (local address 127.0.0.1:44324)
0929 10:30:53.170632 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:53.170708 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:42711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 10:30:53.171788 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.171563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:42711 (local address 127.0.0.1:44336)
0929 10:30:53.171671 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:53.171724 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:42711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 10:30:53.171949 1747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.92.60:42711: Network error: Client connection negotiation failed: client connection to 127.20.92.60:42711: connect: Connection refused (error 111)
W20260929 10:30:53.172036 1829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:30:53.172363 1829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:30:53.172459 1829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:30:53.176702 1829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:30:53.176769 1829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:30:53.176790 1829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:30:53.176808 1829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:30:53.176865 1829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:30:53.177562 1822 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.92.60" port: 42711 } attempt: 2
I20260929 10:30:53.178375 1824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.178093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:42711 (local address 127.0.0.1:44346)
0929 10:30:53.178243 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:53.178307 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:42711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 10:30:53.179073 1824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.178875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:42711 (local address 127.0.0.1:44362)
0929 10:30:53.178976 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:53.179019 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:42711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 10:30:53.179188 1822 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.92.60:42711: Network error: Client connection negotiation failed: client connection to 127.20.92.60:42711: connect: Connection refused (error 111)
I20260929 10:30:53.182804 1829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:46029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:37481,127.20.92.61:40645,127.20.92.60:42711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.60.keytab
--principal=kudu/127.20.92.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.60:42711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.92.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:30:53.183902 1829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:30:53.184921 1829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:30:53.193645 1835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:53.193702 1836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:53.193645 1838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:53.193957 1829 server_base.cc:1034] running on GCE node
I20260929 10:30:53.194490 1829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:30:53.195680 1829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:30:53.196846 1829 hybrid_clock.cc:648] HybridClock initialized: now 1790677853196833 us; error 36 us; skew 500 ppm
Sep 29 10:30:53 dist-test-slave-9cj5 krb5kdc[1666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677853, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:30:53.199688 1829 init.cc:377] Logged in from keytab as kudu/127.20.92.60@KRBTEST.COM (short username kudu)
I20260929 10:30:53.200654 1829 webserver.cc:460] Webserver started at http://127.20.92.60:40957/ using document root <none> and password file <none>
I20260929 10:30:53.201244 1829 fs_manager.cc:359] Metadata directory not provided
I20260929 10:30:53.201305 1829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:30:53.201520 1829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:30:53.203578 1829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-2/data/instance:
uuid: "b8d8937bc34a4ffc9f4d9c6c397d3ccc"
format_stamp: "Formatted at 2026-09-29 10:30:53 on dist-test-slave-9cj5"
server_key: "a0af489bd98fb31a677874eb6824108c"
server_key_iv: "8fa2684910f57c6868d7f9bf023b90f9"
server_key_version: "encryptionkey@0"
I20260929 10:30:53.204154 1829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-2/wal/instance:
uuid: "b8d8937bc34a4ffc9f4d9c6c397d3ccc"
format_stamp: "Formatted at 2026-09-29 10:30:53 on dist-test-slave-9cj5"
server_key: "a0af489bd98fb31a677874eb6824108c"
server_key_iv: "8fa2684910f57c6868d7f9bf023b90f9"
server_key_version: "encryptionkey@0"
I20260929 10:30:53.208068 1829 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:30:53.210693 1845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:53.211997 1829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:30:53.212139 1829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-2/wal
uuid: "b8d8937bc34a4ffc9f4d9c6c397d3ccc"
format_stamp: "Formatted at 2026-09-29 10:30:53 on dist-test-slave-9cj5"
server_key: "a0af489bd98fb31a677874eb6824108c"
server_key_iv: "8fa2684910f57c6868d7f9bf023b90f9"
server_key_version: "encryptionkey@0"
I20260929 10:30:53.212273 1829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:30:53.227787 1829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:30:53.231279 1829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:30:53.231475 1829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:30:53.238925 1829 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.60:42711
I20260929 10:30:53.238951 1897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.60:42711 every 8 connection(s)
I20260929 10:30:53.239907 1829 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-2/data/info.pb
I20260929 10:30:53.241518 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 1829
I20260929 10:30:53.241681 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677679287822-20848-0/minicluster-data/master-2/wal/instance
I20260929 10:30:53.241981 20848 external_mini_cluster.cc:1428] Setting key 8a8562b1f3a599304d525ec1420e3aa6
I20260929 10:30:53.243058 1898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:30:53.247849 1898 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37481 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 10:30:53 dist-test-slave-9cj5 krb5kdc[1666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.60@KRBTEST.COM
Sep 29 10:30:53 dist-test-slave-9cj5 krb5kdc[1666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677853, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.60@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:30:53.258299 1901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.243399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44364 (local address 127.20.92.60:42711)
0929 10:30:53.243822 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:53.243843 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:53.243885 (+ 42us) server_negotiation.cc:407] Connection header received
0929 10:30:53.244743 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:53.244766 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:53.245192 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:53.245563 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:53.246335 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.247816 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.248804 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.249474 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.249602 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:53.252689 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:53.252695 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:53.252708 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:53.252736 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:53.254695 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:53.255208 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:53.255213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:53.255218 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:53.255393 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:53.255675 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:53.255678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:53.255680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:53.256168 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:53.256321 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:53.256685 (+ 364us) server_negotiation.cc:299] Negotiation successful
0929 10:30:53.257013 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":110,"threads_started":1}
I20260929 10:30:53.260035 1751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.250004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54788 (local address 127.20.92.62:37481)
0929 10:30:53.250184 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:53.250191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:53.250589 (+ 398us) server_negotiation.cc:407] Connection header received
0929 10:30:53.250714 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:53.250718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:53.250776 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:53.250871 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:53.251498 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.252290 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.253454 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.253883 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.254010 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:53.256443 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:53.256446 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:53.256448 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:53.256471 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:53.258060 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:53.258651 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:53.258658 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:53.258662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:53.258750 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:53.259069 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:53.259073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:53.259075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:53.259279 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:53.259391 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:53.259751 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 10:30:53.259878 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 10:30:53.260048 1902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.249948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:37481 (local address 127.0.0.1:54788)
0929 10:30:53.250441 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:53.250488 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:53.250611 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:53.250922 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:53.250926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:53.251262 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:30:53.251388 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:53.251394 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.252420 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.252426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:53.253328 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:53.253338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.254000 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.254003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:53.254091 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:53.254577 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:30:53.254589 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:30:53.256286 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:30:53.258194 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:53.258202 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:53.258207 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:53.258531 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:53.258871 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:53.258875 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:53.258877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:53.258962 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:53.259390 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:30:53.259397 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:30:53.259517 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 10:30:53.259671 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 10:30:53.259866 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":205,"threads_started":1}
I20260929 10:30:53.261669 1898 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.92.61" port: 40645 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 10:30:53 dist-test-slave-9cj5 krb5kdc[1666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
Sep 29 10:30:53 dist-test-slave-9cj5 krb5kdc[1666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677853, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.60@KRBTEST.COM for kudu/127.20.92.61@KRBTEST.COM
I20260929 10:30:53.270342 1822 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.92.60" port: 42711 } attempt: 3
I20260929 10:30:53.270851 1751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.260866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54800 (local address 127.20.92.62:37481)
0929 10:30:53.260969 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:53.260973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:53.261186 (+ 213us) server_negotiation.cc:407] Connection header received
0929 10:30:53.261305 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:53.261308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:53.261351 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:53.261433 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:53.262547 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.263093 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.264210 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.264537 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.264644 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:53.267164 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:53.267171 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:53.267177 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:53.267206 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:53.268981 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:53.269552 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:53.269555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:53.269556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:53.269635 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:53.269978 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:53.269980 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:53.269982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:53.270182 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:53.270455 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:53.270547 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 10:30:53.270669 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 10:30:53.271219 1747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.92.60" port: 42711 } attempt: 3
I20260929 10:30:53.272118 1828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.262336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47978 (local address 127.20.92.61:40645)
0929 10:30:53.262482 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:53.262487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:53.263406 (+ 919us) server_negotiation.cc:407] Connection header received
0929 10:30:53.263524 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:53.263526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:53.263570 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:53.263646 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:53.264177 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.264884 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.265637 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.265964 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.266085 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:53.268399 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:53.268403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:53.268405 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:53.268450 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:53.270171 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:53.270702 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:53.270705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:53.270707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:53.270793 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:53.271138 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:53.271142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:53.271143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:53.271372 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:53.271472 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:53.271820 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 10:30:53.271942 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 10:30:53.272634 1902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.262209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:40645 (local address 127.0.0.1:47978)
0929 10:30:53.263312 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:53.263328 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:53.263419 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:53.263690 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:53.263693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:53.263896 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:30:53.264022 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:53.264030 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.264991 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.264996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:53.265516 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:53.265524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.266087 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.266090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:53.266189 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:53.266668 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:30:53.266712 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:30:53.268250 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:30:53.270291 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:53.270294 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:53.270296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:53.270570 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:53.270924 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:53.270927 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:53.270929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:53.271028 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:53.271520 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:30:53.271523 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:30:53.271626 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 10:30:53.272261 (+ 635us) client_negotiation.cc:240] Negotiation successful
0929 10:30:53.272409 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024}
I20260929 10:30:53.273597 1898 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.92.60" port: 42711 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 10:30:53 dist-test-slave-9cj5 krb5kdc[1666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677852, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.61@KRBTEST.COM for kudu/127.20.92.60@KRBTEST.COM
Sep 29 10:30:53 dist-test-slave-9cj5 krb5kdc[1666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677852, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for kudu/127.20.92.60@KRBTEST.COM
Sep 29 10:30:53 dist-test-slave-9cj5 krb5kdc[1666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.61@KRBTEST.COM
I20260929 10:30:53.281587 1901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.271389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44374 (local address 127.20.92.60:42711)
0929 10:30:53.271831 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:53.271836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:53.271850 (+ 14us) server_negotiation.cc:407] Connection header received
0929 10:30:53.271891 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:53.271893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:53.271928 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:53.272030 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:53.272579 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.273325 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.274563 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.274910 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.275019 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:53.277457 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:53.277464 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:53.277468 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:53.277497 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:53.279270 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:53.279785 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:53.279789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:53.279791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:53.279889 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:53.280212 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:53.280217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:53.280219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:53.280462 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:53.280591 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:53.281218 (+ 627us) server_negotiation.cc:299] Negotiation successful
0929 10:30:53.281345 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346}
Sep 29 10:30:53 dist-test-slave-9cj5 krb5kdc[1666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677853, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.60@KRBTEST.COM for kudu/127.20.92.60@KRBTEST.COM
I20260929 10:30:53.282346 1824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.270781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:42711 (local address 127.0.0.1:44374)
0929 10:30:53.271100 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:53.271120 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:53.271227 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:53.272051 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:53.272054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:53.272242 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:30:53.272369 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:53.272378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.273462 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.273469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:53.274388 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:53.274401 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.275029 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.275033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:53.275118 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:53.275642 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:30:53.275657 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:30:53.277295 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:30:53.279387 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:53.279391 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:53.279395 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:53.279662 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:53.279997 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:53.280003 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:53.280008 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:53.280097 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:53.280643 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:30:53.280647 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:30:53.280742 (+ 95us) client_negotiation.cc:769] Sending connection context
0929 10:30:53.280870 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 10:30:53.282142 (+ 1272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219}
I20260929 10:30:53.284065 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.271525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:42711 (local address 127.0.0.1:44380)
0929 10:30:53.271634 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:53.271650 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:53.271726 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:53.273106 (+ 1380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:53.273109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:53.273290 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:30:53.273418 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:53.273428 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.275216 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.275220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:53.276039 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:53.276054 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.276689 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.276693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:53.276781 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:53.277250 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:30:53.277267 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:30:53.278861 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:30:53.282016 (+ 3155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:53.282020 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:53.282022 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:53.282309 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:53.282873 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:53.282877 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:53.282879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:53.282997 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:53.283473 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:30:53.283478 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:30:53.283582 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 10:30:53.283719 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 10:30:53.283878 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 10:30:53.284085 1903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.272390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44380 (local address 127.20.92.60:42711)
0929 10:30:53.272834 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:53.272844 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:53.272858 (+ 14us) server_negotiation.cc:407] Connection header received
0929 10:30:53.272893 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:53.272896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:53.272936 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:53.273535 (+ 599us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:53.273626 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.275076 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.276192 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.276546 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.276999 (+ 453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:53.279036 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:53.279043 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:53.279047 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:53.279077 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:53.281854 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:53.282645 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:53.282651 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:53.282654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:53.282781 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:53.283098 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:53.283102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:53.283106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:53.283355 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:53.283507 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:53.283774 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 10:30:53.283893 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":379,"spinlock_wait_cycles":8832,"thread_start_us":102,"threads_started":1}
I20260929 10:30:53.286451 1902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.274265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:42711 (local address 127.0.0.1:44396)
0929 10:30:53.274740 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:53.274753 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:53.274815 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:53.275621 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:53.275624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:53.275817 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:30:53.275895 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:53.275902 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.276929 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.276935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:53.277828 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:53.277843 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.279881 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.279888 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:53.280013 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:53.280615 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:30:53.280633 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:30:53.282597 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:30:53.284896 (+ 2299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:53.284899 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:53.284900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:53.285128 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:53.285420 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:53.285423 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:53.285424 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:53.285526 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:53.285994 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:30:53.285996 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:30:53.286070 (+ 74us) client_negotiation.cc:769] Sending connection context
0929 10:30:53.286160 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 10:30:53.286300 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":19,"thread_start_us":97,"threads_started":1}
I20260929 10:30:53.286698 1822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b: Bootstrap starting.
I20260929 10:30:53.287258 1747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9867c91722cd4f8d8489ec186c912bfd: Bootstrap starting.
I20260929 10:30:53.287312 1905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.274526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44396 (local address 127.20.92.60:42711)
0929 10:30:53.275278 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:53.275291 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:53.275303 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:30:53.275337 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:53.275340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:53.275380 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:53.275468 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:53.276018 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.276805 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.278284 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.279642 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.279756 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:53.282728 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:53.282731 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:53.282733 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:53.282764 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:53.284781 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:53.285252 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:53.285255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:53.285256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:53.285334 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:53.285619 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:53.285621 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:53.285623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:53.285868 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:53.285961 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:53.286999 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0929 10:30:53.287113 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"spinlock_wait_cycles":1408,"thread_start_us":81,"threads_started":1}
I20260929 10:30:53.287621 1828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.272401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47984 (local address 127.20.92.61:40645)
0929 10:30:53.272637 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:53.272641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:53.272757 (+ 116us) server_negotiation.cc:407] Connection header received
0929 10:30:53.272921 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:53.272926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:53.272974 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:53.273025 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:53.273946 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.274630 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.276044 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.276389 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.276462 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:53.279805 (+ 3343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:53.279809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:53.279811 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:53.279836 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:53.281699 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:53.285875 (+ 4176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:53.285881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:53.285883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:53.286006 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:53.286412 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:53.286417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:53.286418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:53.286682 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:53.286859 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:53.287341 (+ 482us) server_negotiation.cc:299] Negotiation successful
0929 10:30:53.287467 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260929 10:30:53.289651 1822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b: Neither blocks nor log segments found. Creating new log.
I20260929 10:30:53.290002 1898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8d8937bc34a4ffc9f4d9c6c397d3ccc: Bootstrap starting.
I20260929 10:30:53.291097 1747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9867c91722cd4f8d8489ec186c912bfd: Neither blocks nor log segments found. Creating new log.
I20260929 10:30:53.291405 1822 log.cc:824] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b: Log is configured to *not* fsync() on all Append() calls
I20260929 10:30:53.291664 20848 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:30:53.292351 1898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8d8937bc34a4ffc9f4d9c6c397d3ccc: Neither blocks nor log segments found. Creating new log.
I20260929 10:30:53.293123 1898 log.cc:824] T 00000000000000000000000000000000 P b8d8937bc34a4ffc9f4d9c6c397d3ccc: Log is configured to *not* fsync() on all Append() calls
I20260929 10:30:53.293820 1822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b: No bootstrap required, opened a new log
I20260929 10:30:53.294231 1747 log.cc:824] T 00000000000000000000000000000000 P 9867c91722cd4f8d8489ec186c912bfd: Log is configured to *not* fsync() on all Append() calls
I20260929 10:30:53.295964 1898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8d8937bc34a4ffc9f4d9c6c397d3ccc: No bootstrap required, opened a new log
I20260929 10:30:53.298584 1822 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9867c91722cd4f8d8489ec186c912bfd" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37481 } } peers { permanent_uuid: "0fc62a11dea9440895fe42e57967c29b" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 40645 } } peers { permanent_uuid: "b8d8937bc34a4ffc9f4d9c6c397d3ccc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 42711 } }
I20260929 10:30:53.298949 1822 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:30:53.299049 1822 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fc62a11dea9440895fe42e57967c29b, State: Initialized, Role: FOLLOWER
I20260929 10:30:53.299649 1822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9867c91722cd4f8d8489ec186c912bfd" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37481 } } peers { permanent_uuid: "0fc62a11dea9440895fe42e57967c29b" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 40645 } } peers { permanent_uuid: "b8d8937bc34a4ffc9f4d9c6c397d3ccc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 42711 } }
I20260929 10:30:53.299789 1747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9867c91722cd4f8d8489ec186c912bfd: No bootstrap required, opened a new log
I20260929 10:30:53.300854 1915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9867c91722cd4f8d8489ec186c912bfd" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37481 } } peers { permanent_uuid: "0fc62a11dea9440895fe42e57967c29b" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 40645 } } peers { permanent_uuid: "b8d8937bc34a4ffc9f4d9c6c397d3ccc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 42711 } } }
I20260929 10:30:53.301062 1822 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:30:53.301116 1915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:30:53.304208 1747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9867c91722cd4f8d8489ec186c912bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9867c91722cd4f8d8489ec186c912bfd" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37481 } } peers { permanent_uuid: "0fc62a11dea9440895fe42e57967c29b" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 40645 } } peers { permanent_uuid: "b8d8937bc34a4ffc9f4d9c6c397d3ccc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 42711 } }
I20260929 10:30:53.304771 1751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.294067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54810 (local address 127.20.92.62:37481)
0929 10:30:53.294215 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:53.294222 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:53.294942 (+ 720us) server_negotiation.cc:407] Connection header received
0929 10:30:53.295092 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:53.295095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:53.295155 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:53.295244 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:53.296205 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.296737 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.298613 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.298929 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.299053 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:53.300680 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:53.300685 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:53.300690 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:53.300718 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:53.302351 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:53.302879 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:53.302883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:53.302885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:53.302975 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:53.303287 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:53.303291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:53.303293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:53.303509 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:53.303623 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:53.304389 (+ 766us) server_negotiation.cc:299] Negotiation successful
0929 10:30:53.304534 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 10:30:53.305927 1898 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8d8937bc34a4ffc9f4d9c6c397d3ccc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9867c91722cd4f8d8489ec186c912bfd" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37481 } } peers { permanent_uuid: "0fc62a11dea9440895fe42e57967c29b" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 40645 } } peers { permanent_uuid: "b8d8937bc34a4ffc9f4d9c6c397d3ccc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 42711 } }
I20260929 10:30:53.304607 1747 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9867c91722cd4f8d8489ec186c912bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:30:53.306214 1747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9867c91722cd4f8d8489ec186c912bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9867c91722cd4f8d8489ec186c912bfd, State: Initialized, Role: FOLLOWER
I20260929 10:30:53.306280 1898 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8d8937bc34a4ffc9f4d9c6c397d3ccc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:30:53.306397 1898 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8d8937bc34a4ffc9f4d9c6c397d3ccc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8d8937bc34a4ffc9f4d9c6c397d3ccc, State: Initialized, Role: FOLLOWER
I20260929 10:30:53.306771 1747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9867c91722cd4f8d8489ec186c912bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9867c91722cd4f8d8489ec186c912bfd" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37481 } } peers { permanent_uuid: "0fc62a11dea9440895fe42e57967c29b" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 40645 } } peers { permanent_uuid: "b8d8937bc34a4ffc9f4d9c6c397d3ccc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 42711 } }
I20260929 10:30:53.306895 1898 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8d8937bc34a4ffc9f4d9c6c397d3ccc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9867c91722cd4f8d8489ec186c912bfd" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37481 } } peers { permanent_uuid: "0fc62a11dea9440895fe42e57967c29b" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 40645 } } peers { permanent_uuid: "b8d8937bc34a4ffc9f4d9c6c397d3ccc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 42711 } }
I20260929 10:30:53.308229 1747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9867c91722cd4f8d8489ec186c912bfd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:30:53.308342 1898 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8d8937bc34a4ffc9f4d9c6c397d3ccc [sys.catalog]: configured and running, proceeding with master startup.
W20260929 10:30:53.308591 1926 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:30:53.308681 1926 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:30:53.310750 1754 master_runner.cc:186] Error getting master registration for 127.20.92.62:37481: OK, instance_id { permanent_uuid: "9867c91722cd4f8d8489ec186c912bfd" instance_seqno: 1790677852803650 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 10:30:53.316094 1754 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 10:30:53.311396 1930 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8d8937bc34a4ffc9f4d9c6c397d3ccc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9867c91722cd4f8d8489ec186c912bfd" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37481 } } peers { permanent_uuid: "0fc62a11dea9440895fe42e57967c29b" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 40645 } } peers { permanent_uuid: "b8d8937bc34a4ffc9f4d9c6c397d3ccc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 42711 } } }
I20260929 10:30:53.316340 1930 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8d8937bc34a4ffc9f4d9c6c397d3ccc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:30:53.319135 1829 master_runner.cc:263] No leader master found from master b8d8937bc34a4ffc9f4d9c6c397d3ccc
I20260929 10:30:53.319221 1829 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 10:30:53.319353 1679 master_runner.cc:263] No leader master found from master 9867c91722cd4f8d8489ec186c912bfd
I20260929 10:30:53.319404 1905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.304127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44408 (local address 127.20.92.60:42711)
0929 10:30:53.304284 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:53.304288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:53.305210 (+ 922us) server_negotiation.cc:407] Connection header received
0929 10:30:53.308914 (+ 3704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:53.308918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:53.308975 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:53.309028 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:53.310068 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.310923 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.311996 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.312377 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.312486 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:53.314306 (+ 1820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:53.314311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:53.314314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:53.314341 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:53.316159 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:53.317727 (+ 1568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:53.317732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:53.317736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:53.317863 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:53.318229 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:53.318238 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:53.318241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:53.318513 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:53.318665 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:53.319102 (+ 437us) server_negotiation.cc:299] Negotiation successful
0929 10:30:53.319238 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 10:30:53.319419 1679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 10:30:53.319386 1915 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:30:53.319692 1915 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9867c91722cd4f8d8489ec186c912bfd" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37481 } } peers { permanent_uuid: "0fc62a11dea9440895fe42e57967c29b" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 40645 } } peers { permanent_uuid: "b8d8937bc34a4ffc9f4d9c6c397d3ccc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 42711 } }
I20260929 10:30:53.320843 1915 leader_election.cc:288] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9867c91722cd4f8d8489ec186c912bfd (127.20.92.62:37481), b8d8937bc34a4ffc9f4d9c6c397d3ccc (127.20.92.60:42711)
I20260929 10:30:53.321281 1721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0fc62a11dea9440895fe42e57967c29b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9867c91722cd4f8d8489ec186c912bfd" is_pre_election: true
I20260929 10:30:53.321348 1872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0fc62a11dea9440895fe42e57967c29b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8d8937bc34a4ffc9f4d9c6c397d3ccc" is_pre_election: true
I20260929 10:30:53.321614 1721 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9867c91722cd4f8d8489ec186c912bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fc62a11dea9440895fe42e57967c29b in term 0.
I20260929 10:30:53.321636 1872 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b8d8937bc34a4ffc9f4d9c6c397d3ccc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fc62a11dea9440895fe42e57967c29b in term 0.
I20260929 10:30:53.322122 1771 leader_election.cc:302] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0fc62a11dea9440895fe42e57967c29b, 9867c91722cd4f8d8489ec186c912bfd, b8d8937bc34a4ffc9f4d9c6c397d3ccc; no voters:
I20260929 10:30:53.322932 1828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:53.294537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47988 (local address 127.20.92.61:40645)
0929 10:30:53.294654 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:53.294660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:53.304806 (+ 10146us) server_negotiation.cc:407] Connection header received
0929 10:30:53.304981 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:53.304984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:53.305039 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:53.305092 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:53.306089 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.306842 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.307952 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:53.308250 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:53.308328 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:53.317359 (+ 9031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:53.317363 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:53.317366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:53.317395 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:53.319172 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:53.321491 (+ 2319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:53.321495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:53.321497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:53.321602 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:53.321984 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:53.321987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:53.321989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:53.322185 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:53.322322 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:53.322651 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 10:30:53.322777 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 10:30:53.322916 1915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:30:53.323002 1915 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:30:53.323051 1915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:30:53.324153 1915 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9867c91722cd4f8d8489ec186c912bfd" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37481 } } peers { permanent_uuid: "0fc62a11dea9440895fe42e57967c29b" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 40645 } } peers { permanent_uuid: "b8d8937bc34a4ffc9f4d9c6c397d3ccc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 42711 } }
I20260929 10:30:53.324855 1721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0fc62a11dea9440895fe42e57967c29b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9867c91722cd4f8d8489ec186c912bfd"
I20260929 10:30:53.325004 1721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9867c91722cd4f8d8489ec186c912bfd [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:30:53.326196 1721 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9867c91722cd4f8d8489ec186c912bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fc62a11dea9440895fe42e57967c29b in term 1.
W20260929 10:30:53.326231 1949 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9867c91722cd4f8d8489ec186c912bfd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:30:53.326324 1949 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:30:53.326588 1773 leader_election.cc:302] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fc62a11dea9440895fe42e57967c29b, 9867c91722cd4f8d8489ec186c912bfd; no voters:
I20260929 10:30:53.326545 1929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9867c91722cd4f8d8489ec186c912bfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9867c91722cd4f8d8489ec186c912bfd" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37481 } } peers { permanent_uuid: "0fc62a11dea9440895fe42e57967c29b" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 40645 } } peers { permanent_uuid: "b8d8937bc34a4ffc9f4d9c6c397d3ccc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 42711 } } }
I20260929 10:30:53.326666 1929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9867c91722cd4f8d8489ec186c912bfd [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:30:53.327368 1872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0fc62a11dea9440895fe42e57967c29b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8d8937bc34a4ffc9f4d9c6c397d3ccc"
I20260929 10:30:53.327490 1872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8d8937bc34a4ffc9f4d9c6c397d3ccc [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:30:53.328749 1872 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b8d8937bc34a4ffc9f4d9c6c397d3ccc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fc62a11dea9440895fe42e57967c29b in term 1.
W20260929 10:30:53.328976 1952 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b8d8937bc34a4ffc9f4d9c6c397d3ccc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:30:53.329079 1952 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:30:53.329231 1915 leader_election.cc:288] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b [CANDIDATE]: Term 1 election: Requested vote from peers 9867c91722cd4f8d8489ec186c912bfd (127.20.92.62:37481), b8d8937bc34a4ffc9f4d9c6c397d3ccc (127.20.92.60:42711)
I20260929 10:30:53.329448 1915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:30:53.329589 1915 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b [term 1 LEADER]: Becoming Leader. State: Replica: 0fc62a11dea9440895fe42e57967c29b, State: Running, Role: LEADER
I20260929 10:30:53.329996 1915 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9867c91722cd4f8d8489ec186c912bfd" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37481 } } peers { permanent_uuid: "0fc62a11dea9440895fe42e57967c29b" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 40645 } } peers { permanent_uuid: "b8d8937bc34a4ffc9f4d9c6c397d3ccc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 42711 } }
I20260929 10:30:53.331841 1915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fc62a11dea9440895fe42e57967c29b. Latest consensus state: current_term: 1 leader_uuid: "0fc62a11dea9440895fe42e57967c29b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9867c91722cd4f8d8489ec186c912bfd" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37481 } } peers { permanent_uuid: "0fc62a11dea9440895fe42e57967c29b" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 40645 } } peers { permanent_uuid: "b8d8937bc34a4ffc9f4d9c6c397d3ccc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 42711 } } }
I20260929 10:30:53.332015 1915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b [sys.catalog]: This master's current role is: LEADER
I20260929 10:30:53.340364 1957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:30:53.341478 1957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:30:53.353813 1872 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b8d8937bc34a4ffc9f4d9c6c397d3ccc [term 1 FOLLOWER]: Refusing update from remote peer 0fc62a11dea9440895fe42e57967c29b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:30:53.357100 1954 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8d8937bc34a4ffc9f4d9c6c397d3ccc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 42711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:30:53.362047 1930 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8d8937bc34a4ffc9f4d9c6c397d3ccc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fc62a11dea9440895fe42e57967c29b. Latest consensus state: current_term: 1 leader_uuid: "0fc62a11dea9440895fe42e57967c29b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9867c91722cd4f8d8489ec186c912bfd" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37481 } } peers { permanent_uuid: "0fc62a11dea9440895fe42e57967c29b" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 40645 } } peers { permanent_uuid: "b8d8937bc34a4ffc9f4d9c6c397d3ccc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 42711 } } }
I20260929 10:30:53.362217 1930 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8d8937bc34a4ffc9f4d9c6c397d3ccc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:30:53.363636 1721 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9867c91722cd4f8d8489ec186c912bfd [term 1 FOLLOWER]: Refusing update from remote peer 0fc62a11dea9440895fe42e57967c29b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:30:53.366989 1954 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9867c91722cd4f8d8489ec186c912bfd" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:30:53.371845 1929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9867c91722cd4f8d8489ec186c912bfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fc62a11dea9440895fe42e57967c29b. Latest consensus state: current_term: 1 leader_uuid: "0fc62a11dea9440895fe42e57967c29b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9867c91722cd4f8d8489ec186c912bfd" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37481 } } peers { permanent_uuid: "0fc62a11dea9440895fe42e57967c29b" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 40645 } } peers { permanent_uuid: "b8d8937bc34a4ffc9f4d9c6c397d3ccc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 42711 } } }
I20260929 10:30:53.372010 1929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9867c91722cd4f8d8489ec186c912bfd [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:30:53.375006 1930 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8d8937bc34a4ffc9f4d9c6c397d3ccc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0fc62a11dea9440895fe42e57967c29b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9867c91722cd4f8d8489ec186c912bfd" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37481 } } peers { permanent_uuid: "0fc62a11dea9440895fe42e57967c29b" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 40645 } } peers { permanent_uuid: "b8d8937bc34a4ffc9f4d9c6c397d3ccc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 42711 } } }
I20260929 10:30:53.375139 1930 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8d8937bc34a4ffc9f4d9c6c397d3ccc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:30:53.376911 1954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0fc62a11dea9440895fe42e57967c29b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9867c91722cd4f8d8489ec186c912bfd" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37481 } } peers { permanent_uuid: "0fc62a11dea9440895fe42e57967c29b" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 40645 } } peers { permanent_uuid: "b8d8937bc34a4ffc9f4d9c6c397d3ccc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 42711 } } }
I20260929 10:30:53.377085 1954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b [sys.catalog]: This master's current role is: LEADER
I20260929 10:30:53.377367 1958 mvcc.cc:205] Tried to move back new op lower bound from 7334616487321972736 to 7334616487243030528. Current Snapshot: MvccSnapshot[applied={T|T < 7334616487321972736}]
I20260929 10:30:53.382129 1954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0fc62a11dea9440895fe42e57967c29b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9867c91722cd4f8d8489ec186c912bfd" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37481 } } peers { permanent_uuid: "0fc62a11dea9440895fe42e57967c29b" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 40645 } } peers { permanent_uuid: "b8d8937bc34a4ffc9f4d9c6c397d3ccc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 42711 } } }
I20260929 10:30:53.382282 1954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b [sys.catalog]: This master's current role is: LEADER
I20260929 10:30:53.382743 1957 catalog_manager.cc:1347] Generated new cluster ID: ab6f932058314a52b6e7c262ef743010
I20260929 10:30:53.382807 1957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:30:53.384479 1961 mvcc.cc:205] Tried to move back new op lower bound from 7334616487321972736 to 7334616487243030528. Current Snapshot: MvccSnapshot[applied={T|T < 7334616487321972736}]
I20260929 10:30:53.384665 1929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9867c91722cd4f8d8489ec186c912bfd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0fc62a11dea9440895fe42e57967c29b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9867c91722cd4f8d8489ec186c912bfd" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 37481 } } peers { permanent_uuid: "0fc62a11dea9440895fe42e57967c29b" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 40645 } } peers { permanent_uuid: "b8d8937bc34a4ffc9f4d9c6c397d3ccc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 42711 } } }
I20260929 10:30:53.384773 1929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9867c91722cd4f8d8489ec186c912bfd [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:30:53.406338 1957 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:30:53.407761 1957 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:30:53.432152 1957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fc62a11dea9440895fe42e57967c29b: Generated new TSK 0
I20260929 10:30:53.433285 1957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:30:54.327651 1949 catalog_manager.cc:1259] Loaded cluster ID: ab6f932058314a52b6e7c262ef743010
I20260929 10:30:54.327744 1949 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9867c91722cd4f8d8489ec186c912bfd: loading cluster ID for follower catalog manager: success
I20260929 10:30:54.329422 1949 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9867c91722cd4f8d8489ec186c912bfd: acquiring CA information for follower catalog manager: success
I20260929 10:30:54.330296 1952 catalog_manager.cc:1259] Loaded cluster ID: ab6f932058314a52b6e7c262ef743010
I20260929 10:30:54.330353 1952 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b8d8937bc34a4ffc9f4d9c6c397d3ccc: loading cluster ID for follower catalog manager: success
I20260929 10:30:54.330714 1949 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9867c91722cd4f8d8489ec186c912bfd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 10:30:54.331889 1952 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b8d8937bc34a4ffc9f4d9c6c397d3ccc: acquiring CA information for follower catalog manager: success
I20260929 10:30:54.332921 1952 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b8d8937bc34a4ffc9f4d9c6c397d3ccc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 10:30:54.506975 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 1679
I20260929 10:30:54.513662 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 1754
I20260929 10:30:54.519544 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 1829
2026-09-29T10:30:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3141 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:30:54 dist-test-slave-9cj5 krb5kdc[1973](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:30:54 dist-test-slave-9cj5 krb5kdc[1973](info): set up 2 sockets
Sep 29 10:30:54 dist-test-slave-9cj5 krb5kdc[1973](info): commencing operation
krb5kdc: starting...
W20260929 10:30:55.562686 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:30:55 dist-test-slave-9cj5 krb5kdc[1973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:30:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:30:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
I20260929 10:30:55.689556 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:35449
--webserver_interface=127.20.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:45341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:35449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:30:55.803540 1986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:30:55.803992 1986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:30:55.804080 1986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:30:55.809441 1986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:30:55.809516 1986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:30:55.809537 1986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:30:55.809554 1986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:30:55.809571 1986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:30:55.815168 1986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:45341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:35449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:35449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:30:55.816219 1986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:30:55.817185 1986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:30:55.825842 1991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:55.825846 1992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:55.825847 1994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:55.825883 1986 server_base.cc:1034] running on GCE node
I20260929 10:30:55.826687 1986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:30:55.828079 1986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:30:55.829242 1986 hybrid_clock.cc:648] HybridClock initialized: now 1790677855829224 us; error 33 us; skew 500 ppm
Sep 29 10:30:55 dist-test-slave-9cj5 krb5kdc[1973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677855, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:30:55.832013 1986 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:30:55.832942 1986 webserver.cc:460] Webserver started at http://127.20.92.62:36995/ using document root <none> and password file <none>
I20260929 10:30:55.833504 1986 fs_manager.cc:359] Metadata directory not provided
I20260929 10:30:55.833599 1986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:30:55.833933 1986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:30:55.835942 1986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "5ab58239ac1243e68028655496b907f3"
format_stamp: "Formatted at 2026-09-29 10:30:55 on dist-test-slave-9cj5"
server_key: "692572b417b6c865b190f3fff9f727d3"
server_key_iv: "0481194cc934bf09be07bb04c6c9a496"
server_key_version: "encryptionkey@0"
I20260929 10:30:55.836501 1986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "5ab58239ac1243e68028655496b907f3"
format_stamp: "Formatted at 2026-09-29 10:30:55 on dist-test-slave-9cj5"
server_key: "692572b417b6c865b190f3fff9f727d3"
server_key_iv: "0481194cc934bf09be07bb04c6c9a496"
server_key_version: "encryptionkey@0"
I20260929 10:30:55.840260 1986 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 10:30:55.842804 2001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:55.843951 1986 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:30:55.844101 1986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "5ab58239ac1243e68028655496b907f3"
format_stamp: "Formatted at 2026-09-29 10:30:55 on dist-test-slave-9cj5"
server_key: "692572b417b6c865b190f3fff9f727d3"
server_key_iv: "0481194cc934bf09be07bb04c6c9a496"
server_key_version: "encryptionkey@0"
I20260929 10:30:55.844224 1986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:30:55.854086 1986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:30:55.857107 1986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:30:55.857340 1986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:30:55.864552 1986 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:35449
I20260929 10:30:55.864568 2053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:35449 every 8 connection(s)
I20260929 10:30:55.865501 1986 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:30:55.866266 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 1986
I20260929 10:30:55.866401 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:30:55.866680 20848 external_mini_cluster.cc:1428] Setting key 430f589e3d9ce24f9bbad9d5d3dd0df9
I20260929 10:30:55.868883 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:30:55.874608 2054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ab58239ac1243e68028655496b907f3: Bootstrap starting.
Sep 29 10:30:55 dist-test-slave-9cj5 krb5kdc[1973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:30:55.877353 2054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ab58239ac1243e68028655496b907f3: Neither blocks nor log segments found. Creating new log.
I20260929 10:30:55.878181 2054 log.cc:824] T 00000000000000000000000000000000 P 5ab58239ac1243e68028655496b907f3: Log is configured to *not* fsync() on all Append() calls
I20260929 10:30:55.880556 2054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ab58239ac1243e68028655496b907f3: No bootstrap required, opened a new log
I20260929 10:30:55.881364 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:55.868174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32928 (local address 127.20.92.62:35449)
0929 10:30:55.868625 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:55.868639 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:55.868681 (+ 42us) server_negotiation.cc:407] Connection header received
0929 10:30:55.869393 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:55.869410 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:55.869735 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:55.870135 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:55.870935 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:55.871935 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:55.872808 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:55.873275 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:55.873446 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:55.876527 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:55.876532 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:55.876544 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:55.876570 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:55.878437 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:55.878941 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:55.878949 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:55.878955 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:55.879088 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:55.879401 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:55.879403 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:55.879405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:55.879819 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:55.879957 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:55.880300 (+ 343us) server_negotiation.cc:299] Negotiation successful
0929 10:30:55.880575 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":149,"threads_started":1}
I20260929 10:30:55.884198 20848 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:30:55.884446 2054 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ab58239ac1243e68028655496b907f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab58239ac1243e68028655496b907f3" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 35449 } }
I20260929 10:30:55.884707 2054 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ab58239ac1243e68028655496b907f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:30:55.884800 2054 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ab58239ac1243e68028655496b907f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ab58239ac1243e68028655496b907f3, State: Initialized, Role: FOLLOWER
I20260929 10:30:55.885311 2054 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ab58239ac1243e68028655496b907f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab58239ac1243e68028655496b907f3" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 35449 } }
I20260929 10:30:55.885484 2054 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ab58239ac1243e68028655496b907f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:30:55.885582 2054 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ab58239ac1243e68028655496b907f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:30:55.885694 2054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ab58239ac1243e68028655496b907f3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:30:55.886709 2054 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ab58239ac1243e68028655496b907f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab58239ac1243e68028655496b907f3" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 35449 } }
I20260929 10:30:55.887082 2054 leader_election.cc:302] T 00000000000000000000000000000000 P 5ab58239ac1243e68028655496b907f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ab58239ac1243e68028655496b907f3; no voters:
I20260929 10:30:55.887457 2054 leader_election.cc:288] T 00000000000000000000000000000000 P 5ab58239ac1243e68028655496b907f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:30:55.887574 2059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ab58239ac1243e68028655496b907f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:30:55.887823 2059 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ab58239ac1243e68028655496b907f3 [term 1 LEADER]: Becoming Leader. State: Replica: 5ab58239ac1243e68028655496b907f3, State: Running, Role: LEADER
I20260929 10:30:55.888154 2059 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ab58239ac1243e68028655496b907f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab58239ac1243e68028655496b907f3" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 35449 } }
I20260929 10:30:55.888756 2054 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ab58239ac1243e68028655496b907f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:30:55.889974 2061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ab58239ac1243e68028655496b907f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ab58239ac1243e68028655496b907f3. Latest consensus state: current_term: 1 leader_uuid: "5ab58239ac1243e68028655496b907f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab58239ac1243e68028655496b907f3" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 35449 } } }
I20260929 10:30:55.890141 2061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ab58239ac1243e68028655496b907f3 [sys.catalog]: This master's current role is: LEADER
I20260929 10:30:55.890419 2060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ab58239ac1243e68028655496b907f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ab58239ac1243e68028655496b907f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab58239ac1243e68028655496b907f3" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 35449 } } }
I20260929 10:30:55.890529 2060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ab58239ac1243e68028655496b907f3 [sys.catalog]: This master's current role is: LEADER
I20260929 10:30:55.890533 2068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:30:55.893303 2068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:30:55.899364 2068 catalog_manager.cc:1347] Generated new cluster ID: 3879ee9db8a2412cab15496a6489bc57
I20260929 10:30:55.899454 2068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:30:55.913642 2068 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:30:55.914685 2068 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:30:55.919092 2068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ab58239ac1243e68028655496b907f3: Generated new TSK 0
I20260929 10:30:55.919793 2068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:30:55.940234 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 1986
2026-09-29T10:30:55Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1418 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:30:55 dist-test-slave-9cj5 krb5kdc[2081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:30:55 dist-test-slave-9cj5 krb5kdc[2081](info): set up 2 sockets
Sep 29 10:30:55 dist-test-slave-9cj5 krb5kdc[2081](info): commencing operation
krb5kdc: starting...
W20260929 10:30:56.978690 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:30:57 dist-test-slave-9cj5 krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:30:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:30:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
I20260929 10:30:57.109359 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:45535
--webserver_interface=127.20.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:42149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:45535
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:30:57.225133 2094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:30:57.225459 2094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:30:57.225533 2094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:30:57.229678 2094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:30:57.229813 2094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:30:57.229863 2094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:30:57.229916 2094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:30:57.229955 2094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:30:57.235445 2094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:42149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:45535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:45535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:30:57.236624 2094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:30:57.237680 2094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:30:57.246424 2099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:57.246512 2102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:57.246488 2094 server_base.cc:1034] running on GCE node
W20260929 10:30:57.246654 2100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:57.247360 2094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:30:57.248472 2094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:30:57.249644 2094 hybrid_clock.cc:648] HybridClock initialized: now 1790677857249627 us; error 33 us; skew 500 ppm
Sep 29 10:30:57 dist-test-slave-9cj5 krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677857, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:30:57.256758 2094 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:30:57.257786 2094 webserver.cc:460] Webserver started at http://127.20.92.62:37651/ using document root <none> and password file <none>
I20260929 10:30:57.258385 2094 fs_manager.cc:359] Metadata directory not provided
I20260929 10:30:57.258458 2094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:30:57.258713 2094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:30:57.260825 2094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "6ba76be930574b13a58cb1c54c7bfb17"
format_stamp: "Formatted at 2026-09-29 10:30:57 on dist-test-slave-9cj5"
server_key: "b3e58075d162ef95de99a4a377591629"
server_key_iv: "d1d1505ff10e3d5c2b7cd954e8611c3d"
server_key_version: "encryptionkey@0"
I20260929 10:30:57.261399 2094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "6ba76be930574b13a58cb1c54c7bfb17"
format_stamp: "Formatted at 2026-09-29 10:30:57 on dist-test-slave-9cj5"
server_key: "b3e58075d162ef95de99a4a377591629"
server_key_iv: "d1d1505ff10e3d5c2b7cd954e8611c3d"
server_key_version: "encryptionkey@0"
I20260929 10:30:57.265214 2094 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 10:30:57.267788 2109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:57.269114 2094 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:30:57.269273 2094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "6ba76be930574b13a58cb1c54c7bfb17"
format_stamp: "Formatted at 2026-09-29 10:30:57 on dist-test-slave-9cj5"
server_key: "b3e58075d162ef95de99a4a377591629"
server_key_iv: "d1d1505ff10e3d5c2b7cd954e8611c3d"
server_key_version: "encryptionkey@0"
I20260929 10:30:57.269392 2094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:30:57.282694 2094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:30:57.285848 2094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:30:57.286056 2094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:30:57.293462 2094 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:45535
I20260929 10:30:57.293565 2161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:45535 every 8 connection(s)
I20260929 10:30:57.294560 2094 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:30:57.296159 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 2094
I20260929 10:30:57.296320 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:30:57.296610 20848 external_mini_cluster.cc:1428] Setting key 99cfaa5ffb48c5bff4b38e895d733c03
I20260929 10:30:57.297729 2162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:30:57.303437 2162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17: Bootstrap starting.
Sep 29 10:30:57 dist-test-slave-9cj5 krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:30:57.306053 2162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17: Neither blocks nor log segments found. Creating new log.
I20260929 10:30:57.306880 2162 log.cc:824] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17: Log is configured to *not* fsync() on all Append() calls
I20260929 10:30:57.309238 2162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17: No bootstrap required, opened a new log
I20260929 10:30:57.311232 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:57.298014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41634 (local address 127.20.92.62:45535)
0929 10:30:57.298638 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:57.298654 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:57.298691 (+ 37us) server_negotiation.cc:407] Connection header received
0929 10:30:57.299334 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:57.299350 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:57.299679 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:57.300017 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:57.300804 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:57.301919 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:57.302925 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:57.303409 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:57.303607 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:57.306291 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:57.306296 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:57.306309 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:57.306336 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:57.308271 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:57.308782 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:57.308786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:57.308791 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:57.308900 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:57.309206 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:57.309208 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:57.309209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:57.309640 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:57.309876 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:57.310134 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 10:30:57.310437 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":136,"threads_started":1}
I20260929 10:30:57.313306 2162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba76be930574b13a58cb1c54c7bfb17" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45535 } }
I20260929 10:30:57.313575 2162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:30:57.313666 2162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ba76be930574b13a58cb1c54c7bfb17, State: Initialized, Role: FOLLOWER
I20260929 10:30:57.314123 2162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba76be930574b13a58cb1c54c7bfb17" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45535 } }
I20260929 10:30:57.314267 2162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:30:57.314339 2162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:30:57.314461 2162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:30:57.315410 2162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba76be930574b13a58cb1c54c7bfb17" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45535 } }
I20260929 10:30:57.315752 2162 leader_election.cc:302] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ba76be930574b13a58cb1c54c7bfb17; no voters:
I20260929 10:30:57.316107 2162 leader_election.cc:288] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:30:57.316187 2167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:30:57.316413 2167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [term 1 LEADER]: Becoming Leader. State: Replica: 6ba76be930574b13a58cb1c54c7bfb17, State: Running, Role: LEADER
I20260929 10:30:57.316761 2167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba76be930574b13a58cb1c54c7bfb17" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45535 } }
I20260929 10:30:57.317396 2162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:30:57.318578 2169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ba76be930574b13a58cb1c54c7bfb17. Latest consensus state: current_term: 1 leader_uuid: "6ba76be930574b13a58cb1c54c7bfb17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba76be930574b13a58cb1c54c7bfb17" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45535 } } }
I20260929 10:30:57.318603 2168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ba76be930574b13a58cb1c54c7bfb17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba76be930574b13a58cb1c54c7bfb17" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45535 } } }
I20260929 10:30:57.318805 2168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [sys.catalog]: This master's current role is: LEADER
I20260929 10:30:57.318800 2169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [sys.catalog]: This master's current role is: LEADER
I20260929 10:30:57.319236 2176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:30:57.322563 2176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:30:57.328919 2176 catalog_manager.cc:1347] Generated new cluster ID: 18bf720afb1045a98ff4c386f9c130e9
I20260929 10:30:57.329012 2176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:30:57.367301 2176 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:30:57.368379 2176 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:30:57.377470 2176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17: Generated new TSK 0
I20260929 10:30:57.378403 2176 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
I20260929 10:30:57.413980 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:0
--local_ip_for_outbound_sockets=127.20.92.1
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:45535
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:42149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:30:57.528020 2188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:30:57.528303 2188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:30:57.528347 2188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:30:57.532500 2188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:30:57.532572 2188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:30:57.532634 2188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:30:57.538039 2188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:42149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:45535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:30:57.539136 2188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:30:57.540191 2188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:30:57.550248 2194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:57.550323 2188 server_base.cc:1034] running on GCE node
W20260929 10:30:57.550307 2193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:57.550263 2196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:57.551029 2188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:30:57.551628 2188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:30:57.552853 2188 hybrid_clock.cc:648] HybridClock initialized: now 1790677857552799 us; error 67 us; skew 500 ppm
Sep 29 10:30:57 dist-test-slave-9cj5 krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677857, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:30:57.556080 2188 init.cc:377] Logged in from keytab as kudu/127.20.92.1@KRBTEST.COM (short username kudu)
I20260929 10:30:57.557086 2188 webserver.cc:460] Webserver started at http://127.20.92.1:42723/ using document root <none> and password file <none>
I20260929 10:30:57.557729 2188 fs_manager.cc:359] Metadata directory not provided
I20260929 10:30:57.557864 2188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:30:57.558117 2188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:30:57.560316 2188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/data/instance:
uuid: "f224cceb75ea443f9e00bac46dc83f00"
format_stamp: "Formatted at 2026-09-29 10:30:57 on dist-test-slave-9cj5"
server_key: "4a39cd1d4b8f2536fbc59dee176b280c"
server_key_iv: "1af9113c9b11d67036be856f5e211e22"
server_key_version: "encryptionkey@0"
I20260929 10:30:57.560937 2188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance:
uuid: "f224cceb75ea443f9e00bac46dc83f00"
format_stamp: "Formatted at 2026-09-29 10:30:57 on dist-test-slave-9cj5"
server_key: "4a39cd1d4b8f2536fbc59dee176b280c"
server_key_iv: "1af9113c9b11d67036be856f5e211e22"
server_key_version: "encryptionkey@0"
I20260929 10:30:57.565085 2188 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260929 10:30:57.567854 2203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:57.569201 2188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:30:57.569357 2188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "f224cceb75ea443f9e00bac46dc83f00"
format_stamp: "Formatted at 2026-09-29 10:30:57 on dist-test-slave-9cj5"
server_key: "4a39cd1d4b8f2536fbc59dee176b280c"
server_key_iv: "1af9113c9b11d67036be856f5e211e22"
server_key_version: "encryptionkey@0"
I20260929 10:30:57.569480 2188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:30:57.587751 2188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:30:57.591336 2188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:30:57.591598 2188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:30:57.592262 2188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:30:57.593183 2188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:30:57.593264 2188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:57.593344 2188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:30:57.593402 2188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:57.604576 2188 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:39955
I20260929 10:30:57.604598 2316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:39955 every 8 connection(s)
I20260929 10:30:57.605504 2188 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
I20260929 10:30:57.611362 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 2188
I20260929 10:30:57.611478 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance
I20260929 10:30:57.611825 20848 external_mini_cluster.cc:1428] Setting key 6013e73761a50f1cd1efb7c43d410226
Sep 29 10:30:57 dist-test-slave-9cj5 krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677857, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:30:57.620785 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:57.607373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:40715 (local address 127.20.92.62:45535)
0929 10:30:57.607520 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:57.607524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:57.608304 (+ 780us) server_negotiation.cc:407] Connection header received
0929 10:30:57.609603 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:57.609606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:57.609662 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:57.609737 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:57.610960 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:57.611700 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:57.613111 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:57.613421 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:57.613555 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:57.616485 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:57.616489 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:57.616491 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:57.616536 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:57.618444 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:57.619046 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:57.619052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:57.619054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:57.619168 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:57.619554 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:57.619560 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:57.619563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:57.619820 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:57.619927 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:57.620445 (+ 518us) server_negotiation.cc:299] Negotiation successful
0929 10:30:57.620590 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 10:30:57.621610 2319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:57.607672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.1:40715)
0929 10:30:57.608126 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:57.608183 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:57.609407 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:57.609965 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:57.609971 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:57.610334 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:30:57.610799 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:57.610812 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:57.611839 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:57.611845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:57.612962 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:57.612972 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:57.613629 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:57.613635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:57.613854 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:57.614462 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:30:57.614487 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:30:57.616314 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:30:57.618603 (+ 2289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:57.618608 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:57.618620 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:57.618886 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:57.619299 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:57.619309 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:57.619311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:57.619451 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:57.619938 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:30:57.619943 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:30:57.620195 (+ 252us) client_negotiation.cc:769] Sending connection context
0929 10:30:57.620355 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 10:30:57.620625 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.20.92.2@KRBTEST.COM; defaulting to no policy
I20260929 10:30:57.623221 2317 heartbeater.cc:347] Connected to a master server at 127.20.92.62:45535
I20260929 10:30:57.623499 2317 heartbeater.cc:464] Registering TS with master...
I20260929 10:30:57.624109 2317 heartbeater.cc:511] Master 127.20.92.62:45535 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.2@KRBTEST.COM" created.
I20260929 10:30:57.625924 2126 ts_manager.cc:194] Registered new tserver with Master: f224cceb75ea443f9e00bac46dc83f00 (127.20.92.1:39955)
I20260929 10:30:57.627274 2126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.1@KRBTEST.COM'} at 127.20.92.1:40715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
I20260929 10:30:57.640519 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:0
--local_ip_for_outbound_sockets=127.20.92.2
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:45535
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:42149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:30:57.750437 2322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:30:57.750691 2322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:30:57.750736 2322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:30:57.754796 2322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:30:57.754853 2322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:30:57.754914 2322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:30:57.760049 2322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:42149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:45535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:30:57.761008 2322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:30:57.761977 2322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:30:57.771411 2328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:57.771420 2330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:57.771430 2327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:57.771610 2322 server_base.cc:1034] running on GCE node
I20260929 10:30:57.772166 2322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:30:57.772917 2322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:30:57.774102 2322 hybrid_clock.cc:648] HybridClock initialized: now 1790677857774091 us; error 60 us; skew 500 ppm
Sep 29 10:30:57 dist-test-slave-9cj5 krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677857, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:30:57.776830 2322 init.cc:377] Logged in from keytab as kudu/127.20.92.2@KRBTEST.COM (short username kudu)
I20260929 10:30:57.777635 2322 webserver.cc:460] Webserver started at http://127.20.92.2:41047/ using document root <none> and password file <none>
I20260929 10:30:57.778226 2322 fs_manager.cc:359] Metadata directory not provided
I20260929 10:30:57.778288 2322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:30:57.778448 2322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:30:57.780526 2322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/data/instance:
uuid: "a487e45250c24103ae92207d5e7a05de"
format_stamp: "Formatted at 2026-09-29 10:30:57 on dist-test-slave-9cj5"
server_key: "fdf95d4cf95f9c3896a02a1d615a6fd6"
server_key_iv: "616a4df8942a7fde84035efcc2e4c73a"
server_key_version: "encryptionkey@0"
I20260929 10:30:57.781055 2322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance:
uuid: "a487e45250c24103ae92207d5e7a05de"
format_stamp: "Formatted at 2026-09-29 10:30:57 on dist-test-slave-9cj5"
server_key: "fdf95d4cf95f9c3896a02a1d615a6fd6"
server_key_iv: "616a4df8942a7fde84035efcc2e4c73a"
server_key_version: "encryptionkey@0"
I20260929 10:30:57.784967 2322 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 10:30:57.787570 2337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:57.788898 2322 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:30:57.789039 2322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "a487e45250c24103ae92207d5e7a05de"
format_stamp: "Formatted at 2026-09-29 10:30:57 on dist-test-slave-9cj5"
server_key: "fdf95d4cf95f9c3896a02a1d615a6fd6"
server_key_iv: "616a4df8942a7fde84035efcc2e4c73a"
server_key_version: "encryptionkey@0"
I20260929 10:30:57.789153 2322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:30:57.812002 2322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:30:57.815192 2322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:30:57.815402 2322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:30:57.815979 2322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:30:57.816820 2322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:30:57.816864 2322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:57.816927 2322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:30:57.816958 2322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:57.826395 2322 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:42265
I20260929 10:30:57.826421 2450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:42265 every 8 connection(s)
I20260929 10:30:57.827298 2322 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
I20260929 10:30:57.837364 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 2322
I20260929 10:30:57.837499 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance
I20260929 10:30:57.837847 20848 external_mini_cluster.cc:1428] Setting key d7d37766d375b612bc8a00374b7045fc
Sep 29 10:30:57 dist-test-slave-9cj5 krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677857, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:30:57.843116 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:57.829266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:35541 (local address 127.20.92.62:45535)
0929 10:30:57.829408 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:57.829412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:57.830353 (+ 941us) server_negotiation.cc:407] Connection header received
0929 10:30:57.831923 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:57.831927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:57.831999 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:57.832115 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:57.833334 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:57.833992 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:57.835216 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:57.835483 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:57.835607 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:57.838645 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:57.838650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:57.838653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:57.838683 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:57.840452 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:57.841207 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:57.841213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:57.841216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:57.841322 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:57.841724 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:57.841729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:57.841732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:57.842040 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:57.842160 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:57.842728 (+ 568us) server_negotiation.cc:299] Negotiation successful
0929 10:30:57.842876 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 10:30:57.843938 2453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:57.829644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.2:35541)
0929 10:30:57.830188 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:57.830231 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:57.831688 (+ 1457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:57.832291 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:57.832298 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:57.832682 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:30:57.833179 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:57.833192 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:57.834128 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:57.834134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:57.835079 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:57.835088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:57.835649 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:57.835653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:57.835826 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:57.836504 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:30:57.836522 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:30:57.838360 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:30:57.840627 (+ 2267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:57.840636 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:57.840655 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:57.841062 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:57.841429 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:57.841431 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:57.841433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:57.841620 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:57.842179 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:30:57.842184 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:30:57.842440 (+ 256us) client_negotiation.cc:769] Sending connection context
0929 10:30:57.842640 (+ 200us) client_negotiation.cc:240] Negotiation successful
0929 10:30:57.842934 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":142,"threads_started":1}
I20260929 10:30:57.845067 2451 heartbeater.cc:347] Connected to a master server at 127.20.92.62:45535
I20260929 10:30:57.845306 2451 heartbeater.cc:464] Registering TS with master...
I20260929 10:30:57.845858 2451 heartbeater.cc:511] Master 127.20.92.62:45535 requested a full tablet report, sending...
I20260929 10:30:57.846937 2126 ts_manager.cc:194] Registered new tserver with Master: a487e45250c24103ae92207d5e7a05de (127.20.92.2:42265)
I20260929 10:30:57.847640 2126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.2@KRBTEST.COM'} at 127.20.92.2:35541
WARNING: no policy specified for kudu/127.20.92.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
I20260929 10:30:57.865170 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:0
--local_ip_for_outbound_sockets=127.20.92.3
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:45535
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:42149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:30:57.979822 2456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:30:57.980109 2456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:30:57.980186 2456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:30:57.984227 2456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:30:57.984326 2456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:30:57.984417 2456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:30:57.989815 2456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:42149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:45535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:30:57.990914 2456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:30:57.991926 2456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:30:58.002345 2462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:58.002379 2464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:58.002604 2456 server_base.cc:1034] running on GCE node
W20260929 10:30:58.002345 2461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:58.003110 2456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:30:58.003741 2456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:30:58.004897 2456 hybrid_clock.cc:648] HybridClock initialized: now 1790677858004866 us; error 46 us; skew 500 ppm
Sep 29 10:30:58 dist-test-slave-9cj5 krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677858, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:30:58.007804 2456 init.cc:377] Logged in from keytab as kudu/127.20.92.3@KRBTEST.COM (short username kudu)
I20260929 10:30:58.008845 2456 webserver.cc:460] Webserver started at http://127.20.92.3:34939/ using document root <none> and password file <none>
I20260929 10:30:58.009450 2456 fs_manager.cc:359] Metadata directory not provided
I20260929 10:30:58.009516 2456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:30:58.009737 2456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:30:58.011914 2456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/data/instance:
uuid: "eb5d098367f54935b85164679a41e8bc"
format_stamp: "Formatted at 2026-09-29 10:30:58 on dist-test-slave-9cj5"
server_key: "5193a2dfda4eff069a12b3b348e7bd8c"
server_key_iv: "6b2244dd640c4e719dc1bb8ace354704"
server_key_version: "encryptionkey@0"
I20260929 10:30:58.012476 2456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance:
uuid: "eb5d098367f54935b85164679a41e8bc"
format_stamp: "Formatted at 2026-09-29 10:30:58 on dist-test-slave-9cj5"
server_key: "5193a2dfda4eff069a12b3b348e7bd8c"
server_key_iv: "6b2244dd640c4e719dc1bb8ace354704"
server_key_version: "encryptionkey@0"
I20260929 10:30:58.016315 2456 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 10:30:58.019134 2471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:58.020375 2456 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:30:58.020520 2456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "eb5d098367f54935b85164679a41e8bc"
format_stamp: "Formatted at 2026-09-29 10:30:58 on dist-test-slave-9cj5"
server_key: "5193a2dfda4eff069a12b3b348e7bd8c"
server_key_iv: "6b2244dd640c4e719dc1bb8ace354704"
server_key_version: "encryptionkey@0"
I20260929 10:30:58.020644 2456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:30:58.038007 2456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:30:58.041129 2456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:30:58.041352 2456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:30:58.042025 2456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:30:58.042866 2456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:30:58.042938 2456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:58.043009 2456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:30:58.043061 2456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:58.053049 2456 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:33341
I20260929 10:30:58.053071 2584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:33341 every 8 connection(s)
I20260929 10:30:58.054067 2456 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:30:58.062150 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 2456
I20260929 10:30:58.062266 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance
I20260929 10:30:58.062599 20848 external_mini_cluster.cc:1428] Setting key 7bb988f5f064d52cb038999962cd97a6
Sep 29 10:30:58 dist-test-slave-9cj5 krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677858, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:30:58.068357 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:58.055978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:55539 (local address 127.20.92.62:45535)
0929 10:30:58.056158 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:58.056163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:58.056959 (+ 796us) server_negotiation.cc:407] Connection header received
0929 10:30:58.058414 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:58.058417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:58.058467 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:58.058623 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:58.059729 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.060276 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.061465 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.061697 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.061858 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:58.064451 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:58.064455 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:58.064457 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:58.064481 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:58.066054 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:58.066673 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:58.066678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:58.066680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:58.066769 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:58.067153 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:58.067156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:58.067158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:58.067405 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:58.067525 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:58.068049 (+ 524us) server_negotiation.cc:299] Negotiation successful
0929 10:30:58.068174 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 10:30:58.069619 2587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:58.056354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.3:55539)
0929 10:30:58.056800 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:58.056842 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:58.058159 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:58.058716 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:58.058723 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:58.059074 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:30:58.059588 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:58.059601 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.060425 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.060428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:58.061328 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:58.061338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.061911 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.061915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:58.062092 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:58.062770 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:30:58.062787 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:30:58.064270 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:30:58.066187 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:58.066194 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:58.066211 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:58.066549 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:58.066872 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:58.066878 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:58.066880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:58.067036 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:58.067536 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:30:58.067546 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:30:58.067795 (+ 249us) client_negotiation.cc:769] Sending connection context
0929 10:30:58.068003 (+ 208us) client_negotiation.cc:240] Negotiation successful
0929 10:30:58.068363 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":113,"threads_started":1}
I20260929 10:30:58.071017 2585 heartbeater.cc:347] Connected to a master server at 127.20.92.62:45535
I20260929 10:30:58.071254 2585 heartbeater.cc:464] Registering TS with master...
I20260929 10:30:58.071717 2585 heartbeater.cc:511] Master 127.20.92.62:45535 requested a full tablet report, sending...
I20260929 10:30:58.072726 2126 ts_manager.cc:194] Registered new tserver with Master: eb5d098367f54935b85164679a41e8bc (127.20.92.3:33341)
I20260929 10:30:58.073400 2126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.3@KRBTEST.COM'} at 127.20.92.3:55539
I20260929 10:30:58.076468 20848 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:30:58.086006 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:58.078516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41642 (local address 127.20.92.62:45535)
0929 10:30:58.078688 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:58.078693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:58.078833 (+ 140us) server_negotiation.cc:407] Connection header received
0929 10:30:58.078997 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:58.079007 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:58.079048 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:58.079127 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:58.079894 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.080402 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.081252 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.081477 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.081605 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:58.083017 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:58.083029 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:58.083032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:58.083054 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:58.084260 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:58.084684 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:58.084688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:58.084690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:58.084773 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:58.085074 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:58.085079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:58.085081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:58.085303 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:58.085449 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:58.085697 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 10:30:58.085842 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 10:30:58.088934 2126 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:30:58.092016 2126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:30:58.106379 2600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:58.101335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:39955 (local address 127.0.0.1:58516)
0929 10:30:58.101831 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:58.101843 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:58.101924 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:58.102493 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:58.102498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:58.102521 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:30:58.102728 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:58.102735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.103744 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.103749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:58.104961 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:58.104970 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.105799 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.105802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:58.105914 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:58.105950 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 10:30:58.106090 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 10:30:58.106215 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":32,"spinlock_wait_cycles":4352,"thread_start_us":97,"threads_started":1}
I20260929 10:30:58.106380 2601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:58.101739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58516 (local address 127.20.92.1:39955)
0929 10:30:58.102000 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:58.102016 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:58.102042 (+ 26us) server_negotiation.cc:407] Connection header received
0929 10:30:58.102115 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:58.102121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:58.102320 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:58.102486 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:30:58.102865 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.103608 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.105089 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.105651 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.105803 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:58.105956 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:58.106119 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 10:30:58.106226 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":70,"threads_started":1}
I20260929 10:30:58.107321 2599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:58.101032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52610 (local address 127.20.92.3:33341)
0929 10:30:58.102136 (+ 1104us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:58.102146 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:58.102164 (+ 18us) server_negotiation.cc:407] Connection header received
0929 10:30:58.102221 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:58.102225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:58.102361 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:58.102520 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:30:58.102890 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.104043 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.105864 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.106687 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.106787 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:58.106914 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:58.107037 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 10:30:58.107140 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1021,"thread_start_us":129,"threads_started":1}
I20260929 10:30:58.107384 2596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:58.100613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:42265 (local address 127.0.0.1:49186)
0929 10:30:58.101298 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:58.101351 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:58.101498 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:58.102546 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:58.102549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:58.102575 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:30:58.102737 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:58.102746 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.104193 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.104196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:58.105698 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:58.105709 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.106976 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.106979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:58.107070 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:58.107093 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 10:30:58.107170 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 10:30:58.107256 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":259,"spinlock_wait_cycles":768,"thread_start_us":84,"threads_started":1}
I20260929 10:30:58.107733 2597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:58.100746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49186 (local address 127.20.92.2:42265)
0929 10:30:58.102155 (+ 1409us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:58.102165 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:58.102184 (+ 19us) server_negotiation.cc:407] Connection header received
0929 10:30:58.102247 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:58.102251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:58.102398 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:58.102688 (+ 290us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:30:58.102852 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.104033 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.105864 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.106678 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.106787 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:58.106914 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:58.107537 (+ 623us) server_negotiation.cc:299] Negotiation successful
0929 10:30:58.107625 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1299,"thread_start_us":133,"threads_started":1}
I20260929 10:30:58.107995 2598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:58.100834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:33341 (local address 127.0.0.1:52610)
0929 10:30:58.101507 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:58.101521 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:58.101613 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:58.102679 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:58.102684 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:58.102702 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:30:58.102790 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:58.102798 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.104193 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.104196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:58.105698 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:58.105709 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.106806 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.106811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:58.106899 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:58.106932 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 10:30:58.107012 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 10:30:58.107092 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":98,"thread_start_us":95,"threads_started":1}
I20260929 10:30:58.110296 2385 tablet_service.cc:1467] Processing CreateTablet for tablet 67aa1ff0ce4446e29a69a31754191f9b (DEFAULT_TABLE table=test-table [id=c419c4b49e914163a6c95c154a6f03e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:30:58.110140 2250 tablet_service.cc:1467] Processing CreateTablet for tablet 67aa1ff0ce4446e29a69a31754191f9b (DEFAULT_TABLE table=test-table [id=c419c4b49e914163a6c95c154a6f03e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:30:58.110509 2519 tablet_service.cc:1467] Processing CreateTablet for tablet 67aa1ff0ce4446e29a69a31754191f9b (DEFAULT_TABLE table=test-table [id=c419c4b49e914163a6c95c154a6f03e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:30:58.111299 2385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67aa1ff0ce4446e29a69a31754191f9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:30:58.111299 2250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67aa1ff0ce4446e29a69a31754191f9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:30:58.111318 2519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67aa1ff0ce4446e29a69a31754191f9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:30:58.117489 2602 tablet_bootstrap.cc:492] T 67aa1ff0ce4446e29a69a31754191f9b P a487e45250c24103ae92207d5e7a05de: Bootstrap starting.
I20260929 10:30:58.119094 2603 tablet_bootstrap.cc:492] T 67aa1ff0ce4446e29a69a31754191f9b P eb5d098367f54935b85164679a41e8bc: Bootstrap starting.
I20260929 10:30:58.120646 2602 tablet_bootstrap.cc:654] T 67aa1ff0ce4446e29a69a31754191f9b P a487e45250c24103ae92207d5e7a05de: Neither blocks nor log segments found. Creating new log.
I20260929 10:30:58.121344 2603 tablet_bootstrap.cc:654] T 67aa1ff0ce4446e29a69a31754191f9b P eb5d098367f54935b85164679a41e8bc: Neither blocks nor log segments found. Creating new log.
I20260929 10:30:58.121591 2602 log.cc:824] T 67aa1ff0ce4446e29a69a31754191f9b P a487e45250c24103ae92207d5e7a05de: Log is configured to *not* fsync() on all Append() calls
I20260929 10:30:58.122676 2603 log.cc:824] T 67aa1ff0ce4446e29a69a31754191f9b P eb5d098367f54935b85164679a41e8bc: Log is configured to *not* fsync() on all Append() calls
I20260929 10:30:58.122761 2604 tablet_bootstrap.cc:492] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00: Bootstrap starting.
I20260929 10:30:58.124217 2602 tablet_bootstrap.cc:492] T 67aa1ff0ce4446e29a69a31754191f9b P a487e45250c24103ae92207d5e7a05de: No bootstrap required, opened a new log
I20260929 10:30:58.124421 2602 ts_tablet_manager.cc:1397] T 67aa1ff0ce4446e29a69a31754191f9b P a487e45250c24103ae92207d5e7a05de: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 10:30:58.124686 2603 tablet_bootstrap.cc:492] T 67aa1ff0ce4446e29a69a31754191f9b P eb5d098367f54935b85164679a41e8bc: No bootstrap required, opened a new log
I20260929 10:30:58.124868 2603 ts_tablet_manager.cc:1397] T 67aa1ff0ce4446e29a69a31754191f9b P eb5d098367f54935b85164679a41e8bc: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 10:30:58.124922 2604 tablet_bootstrap.cc:654] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00: Neither blocks nor log segments found. Creating new log.
I20260929 10:30:58.125779 2604 log.cc:824] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00: Log is configured to *not* fsync() on all Append() calls
I20260929 10:30:58.127799 2604 tablet_bootstrap.cc:492] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00: No bootstrap required, opened a new log
I20260929 10:30:58.127995 2604 ts_tablet_manager.cc:1397] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 10:30:58.127938 2602 raft_consensus.cc:348] T 67aa1ff0ce4446e29a69a31754191f9b P a487e45250c24103ae92207d5e7a05de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a487e45250c24103ae92207d5e7a05de" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 42265 } } peers { permanent_uuid: "f224cceb75ea443f9e00bac46dc83f00" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39955 } } peers { permanent_uuid: "eb5d098367f54935b85164679a41e8bc" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 33341 } }
I20260929 10:30:58.128242 2602 raft_consensus.cc:374] T 67aa1ff0ce4446e29a69a31754191f9b P a487e45250c24103ae92207d5e7a05de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:30:58.128295 2602 raft_consensus.cc:729] T 67aa1ff0ce4446e29a69a31754191f9b P a487e45250c24103ae92207d5e7a05de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a487e45250c24103ae92207d5e7a05de, State: Initialized, Role: FOLLOWER
I20260929 10:30:58.128638 2603 raft_consensus.cc:348] T 67aa1ff0ce4446e29a69a31754191f9b P eb5d098367f54935b85164679a41e8bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a487e45250c24103ae92207d5e7a05de" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 42265 } } peers { permanent_uuid: "f224cceb75ea443f9e00bac46dc83f00" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39955 } } peers { permanent_uuid: "eb5d098367f54935b85164679a41e8bc" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 33341 } }
I20260929 10:30:58.128815 2602 consensus_queue.cc:260] T 67aa1ff0ce4446e29a69a31754191f9b P a487e45250c24103ae92207d5e7a05de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a487e45250c24103ae92207d5e7a05de" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 42265 } } peers { permanent_uuid: "f224cceb75ea443f9e00bac46dc83f00" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39955 } } peers { permanent_uuid: "eb5d098367f54935b85164679a41e8bc" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 33341 } }
I20260929 10:30:58.128895 2603 raft_consensus.cc:374] T 67aa1ff0ce4446e29a69a31754191f9b P eb5d098367f54935b85164679a41e8bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:30:58.128945 2603 raft_consensus.cc:729] T 67aa1ff0ce4446e29a69a31754191f9b P eb5d098367f54935b85164679a41e8bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb5d098367f54935b85164679a41e8bc, State: Initialized, Role: FOLLOWER
I20260929 10:30:58.129348 2603 consensus_queue.cc:260] T 67aa1ff0ce4446e29a69a31754191f9b P eb5d098367f54935b85164679a41e8bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a487e45250c24103ae92207d5e7a05de" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 42265 } } peers { permanent_uuid: "f224cceb75ea443f9e00bac46dc83f00" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39955 } } peers { permanent_uuid: "eb5d098367f54935b85164679a41e8bc" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 33341 } }
I20260929 10:30:58.129861 2602 ts_tablet_manager.cc:1428] T 67aa1ff0ce4446e29a69a31754191f9b P a487e45250c24103ae92207d5e7a05de: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260929 10:30:58.130170 2585 heartbeater.cc:503] Master 127.20.92.62:45535 was elected leader, sending a full tablet report...
I20260929 10:30:58.130379 2603 ts_tablet_manager.cc:1428] T 67aa1ff0ce4446e29a69a31754191f9b P eb5d098367f54935b85164679a41e8bc: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260929 10:30:58.130586 2451 heartbeater.cc:503] Master 127.20.92.62:45535 was elected leader, sending a full tablet report...
I20260929 10:30:58.134028 2604 raft_consensus.cc:348] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a487e45250c24103ae92207d5e7a05de" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 42265 } } peers { permanent_uuid: "f224cceb75ea443f9e00bac46dc83f00" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39955 } } peers { permanent_uuid: "eb5d098367f54935b85164679a41e8bc" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 33341 } }
I20260929 10:30:58.134330 2604 raft_consensus.cc:374] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:30:58.134392 2604 raft_consensus.cc:729] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f224cceb75ea443f9e00bac46dc83f00, State: Initialized, Role: FOLLOWER
I20260929 10:30:58.134869 2604 consensus_queue.cc:260] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a487e45250c24103ae92207d5e7a05de" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 42265 } } peers { permanent_uuid: "f224cceb75ea443f9e00bac46dc83f00" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39955 } } peers { permanent_uuid: "eb5d098367f54935b85164679a41e8bc" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 33341 } }
I20260929 10:30:58.135604 2317 heartbeater.cc:503] Master 127.20.92.62:45535 was elected leader, sending a full tablet report...
I20260929 10:30:58.135910 2604 ts_tablet_manager.cc:1428] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260929 10:30:58.148202 2610 raft_consensus.cc:482] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:30:58.148449 2610 raft_consensus.cc:504] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a487e45250c24103ae92207d5e7a05de" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 42265 } } peers { permanent_uuid: "f224cceb75ea443f9e00bac46dc83f00" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39955 } } peers { permanent_uuid: "eb5d098367f54935b85164679a41e8bc" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 33341 } }
I20260929 10:30:58.149650 2610 leader_election.cc:288] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a487e45250c24103ae92207d5e7a05de (127.20.92.2:42265), eb5d098367f54935b85164679a41e8bc (127.20.92.3:33341)
I20260929 10:30:58.153573 2599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:58.150033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:36369 (local address 127.20.92.3:33341)
0929 10:30:58.150175 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:58.150180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:58.150382 (+ 202us) server_negotiation.cc:407] Connection header received
0929 10:30:58.150512 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:58.150515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:58.150557 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:58.150614 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:30:58.150954 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.151469 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.152589 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.152986 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.153108 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:58.153175 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:58.153369 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 10:30:58.153441 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 10:30:58.153597 2597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:58.149954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:47991 (local address 127.20.92.2:42265)
0929 10:30:58.150114 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:58.150119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:58.150343 (+ 224us) server_negotiation.cc:407] Connection header received
0929 10:30:58.150506 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:58.150509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:58.150560 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:58.150678 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:30:58.150999 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.151467 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.152602 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.153005 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.153130 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:58.153189 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:58.153411 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 10:30:58.153479 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 10:30:58.153604 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:58.149938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:33341 (local address 127.20.92.1:36369)
0929 10:30:58.150257 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:58.150268 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:58.150400 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:58.150658 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:58.150664 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:58.150682 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:30:58.150840 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:58.150845 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.151587 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.151593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:58.152459 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:58.152469 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.153132 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.153136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:58.153234 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:58.153264 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 10:30:58.153324 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 10:30:58.153414 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":9,"spinlock_wait_cycles":7808,"thread_start_us":111,"threads_started":1}
I20260929 10:30:58.153610 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:58.149844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:42265 (local address 127.20.92.1:47991)
0929 10:30:58.150226 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:58.150255 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:58.150384 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:58.150708 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:58.150714 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:58.150752 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:30:58.150866 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:58.150871 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.151584 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.151590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:58.152460 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:58.152469 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.153147 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.153150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:58.153271 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:58.153288 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 10:30:58.153381 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 10:30:58.153445 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":28,"spinlock_wait_cycles":768,"thread_start_us":137,"threads_started":1}
I20260929 10:30:58.154294 2405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67aa1ff0ce4446e29a69a31754191f9b" candidate_uuid: "f224cceb75ea443f9e00bac46dc83f00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a487e45250c24103ae92207d5e7a05de" is_pre_election: true
I20260929 10:30:58.154317 2539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67aa1ff0ce4446e29a69a31754191f9b" candidate_uuid: "f224cceb75ea443f9e00bac46dc83f00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb5d098367f54935b85164679a41e8bc" is_pre_election: true
I20260929 10:30:58.154621 2539 raft_consensus.cc:2454] T 67aa1ff0ce4446e29a69a31754191f9b P eb5d098367f54935b85164679a41e8bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f224cceb75ea443f9e00bac46dc83f00 in term 0.
I20260929 10:30:58.154630 2405 raft_consensus.cc:2454] T 67aa1ff0ce4446e29a69a31754191f9b P a487e45250c24103ae92207d5e7a05de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f224cceb75ea443f9e00bac46dc83f00 in term 0.
I20260929 10:30:58.155097 2207 leader_election.cc:302] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eb5d098367f54935b85164679a41e8bc, f224cceb75ea443f9e00bac46dc83f00; no voters:
I20260929 10:30:58.155377 2610 raft_consensus.cc:2781] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:30:58.155460 2610 raft_consensus.cc:482] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:30:58.155510 2610 raft_consensus.cc:3037] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:30:58.156525 2610 raft_consensus.cc:504] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a487e45250c24103ae92207d5e7a05de" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 42265 } } peers { permanent_uuid: "f224cceb75ea443f9e00bac46dc83f00" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39955 } } peers { permanent_uuid: "eb5d098367f54935b85164679a41e8bc" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 33341 } }
I20260929 10:30:58.156860 2610 leader_election.cc:288] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00 [CANDIDATE]: Term 1 election: Requested vote from peers a487e45250c24103ae92207d5e7a05de (127.20.92.2:42265), eb5d098367f54935b85164679a41e8bc (127.20.92.3:33341)
I20260929 10:30:58.157173 2405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67aa1ff0ce4446e29a69a31754191f9b" candidate_uuid: "f224cceb75ea443f9e00bac46dc83f00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a487e45250c24103ae92207d5e7a05de"
I20260929 10:30:58.157199 2539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67aa1ff0ce4446e29a69a31754191f9b" candidate_uuid: "f224cceb75ea443f9e00bac46dc83f00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb5d098367f54935b85164679a41e8bc"
I20260929 10:30:58.157301 2405 raft_consensus.cc:3037] T 67aa1ff0ce4446e29a69a31754191f9b P a487e45250c24103ae92207d5e7a05de [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:30:58.157318 2539 raft_consensus.cc:3037] T 67aa1ff0ce4446e29a69a31754191f9b P eb5d098367f54935b85164679a41e8bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:30:58.158438 2405 raft_consensus.cc:2454] T 67aa1ff0ce4446e29a69a31754191f9b P a487e45250c24103ae92207d5e7a05de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f224cceb75ea443f9e00bac46dc83f00 in term 1.
I20260929 10:30:58.158440 2539 raft_consensus.cc:2454] T 67aa1ff0ce4446e29a69a31754191f9b P eb5d098367f54935b85164679a41e8bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f224cceb75ea443f9e00bac46dc83f00 in term 1.
I20260929 10:30:58.158715 2207 leader_election.cc:302] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eb5d098367f54935b85164679a41e8bc, f224cceb75ea443f9e00bac46dc83f00; no voters:
I20260929 10:30:58.158937 2610 raft_consensus.cc:2781] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:30:58.159178 2610 raft_consensus.cc:686] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00 [term 1 LEADER]: Becoming Leader. State: Replica: f224cceb75ea443f9e00bac46dc83f00, State: Running, Role: LEADER
I20260929 10:30:58.159560 2610 consensus_queue.cc:237] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a487e45250c24103ae92207d5e7a05de" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 42265 } } peers { permanent_uuid: "f224cceb75ea443f9e00bac46dc83f00" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39955 } } peers { permanent_uuid: "eb5d098367f54935b85164679a41e8bc" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 33341 } }
I20260929 10:30:58.162927 2126 catalog_manager.cc:5495] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00 reported cstate change: term changed from 0 to 1, leader changed from <none> to f224cceb75ea443f9e00bac46dc83f00 (127.20.92.1). New cstate: current_term: 1 leader_uuid: "f224cceb75ea443f9e00bac46dc83f00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a487e45250c24103ae92207d5e7a05de" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 42265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f224cceb75ea443f9e00bac46dc83f00" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb5d098367f54935b85164679a41e8bc" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 33341 } health_report { overall_health: UNKNOWN } } }
I20260929 10:30:58.185880 2601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:58.182283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58532 (local address 127.20.92.1:39955)
0929 10:30:58.182433 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:58.182438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:58.182452 (+ 14us) server_negotiation.cc:407] Connection header received
0929 10:30:58.182560 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:58.182562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:58.182649 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:58.182768 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:30:58.183010 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.183529 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.184394 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.184626 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.184739 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:58.185050 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:30:58.185461 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:30:58.185605 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:58.185649 (+ 44us) server_negotiation.cc:299] Negotiation successful
0929 10:30:58.185704 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 10:30:58.191847 2539 raft_consensus.cc:1261] T 67aa1ff0ce4446e29a69a31754191f9b P eb5d098367f54935b85164679a41e8bc [term 1 FOLLOWER]: Refusing update from remote peer f224cceb75ea443f9e00bac46dc83f00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:30:58.191847 2405 raft_consensus.cc:1261] T 67aa1ff0ce4446e29a69a31754191f9b P a487e45250c24103ae92207d5e7a05de [term 1 FOLLOWER]: Refusing update from remote peer f224cceb75ea443f9e00bac46dc83f00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:30:58.192641 2613 consensus_queue.cc:1035] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a487e45250c24103ae92207d5e7a05de" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 42265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:30:58.192830 2610 consensus_queue.cc:1035] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5d098367f54935b85164679a41e8bc" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 33341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:30:58.200218 2617 mvcc.cc:205] Tried to move back new op lower bound from 7334616507146833920 to 7334616507024433152. Current Snapshot: MvccSnapshot[applied={T|T < 7334616507146833920}]
I20260929 10:30:58.200799 2618 mvcc.cc:205] Tried to move back new op lower bound from 7334616507146833920 to 7334616507024433152. Current Snapshot: MvccSnapshot[applied={T|T < 7334616507146833920}]
I20260929 10:30:58.210472 2126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:30:58.210770 2126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:30:58.213454 2126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17: Sending DeleteTablet for 3 replicas of tablet 67aa1ff0ce4446e29a69a31754191f9b
I20260929 10:30:58.214349 2250 tablet_service.cc:1514] Processing DeleteTablet for tablet 67aa1ff0ce4446e29a69a31754191f9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:30:58 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:58516
I20260929 10:30:58.214354 2385 tablet_service.cc:1514] Processing DeleteTablet for tablet 67aa1ff0ce4446e29a69a31754191f9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:30:58 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:49186
I20260929 10:30:58.214416 2519 tablet_service.cc:1514] Processing DeleteTablet for tablet 67aa1ff0ce4446e29a69a31754191f9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:30:58 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:52610
I20260929 10:30:58.214872 2627 tablet_replica.cc:330] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00: stopping tablet replica
I20260929 10:30:58.215096 2628 tablet_replica.cc:330] T 67aa1ff0ce4446e29a69a31754191f9b P eb5d098367f54935b85164679a41e8bc: stopping tablet replica
I20260929 10:30:58.215138 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 2188
I20260929 10:30:58.215207 2627 raft_consensus.cc:2229] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:30:58.215907 2626 tablet_replica.cc:330] T 67aa1ff0ce4446e29a69a31754191f9b P a487e45250c24103ae92207d5e7a05de: stopping tablet replica
I20260929 10:30:58.216114 2628 raft_consensus.cc:2229] T 67aa1ff0ce4446e29a69a31754191f9b P eb5d098367f54935b85164679a41e8bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:30:58.216213 2626 raft_consensus.cc:2229] T 67aa1ff0ce4446e29a69a31754191f9b P a487e45250c24103ae92207d5e7a05de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:30:58.216509 2628 raft_consensus.cc:2258] T 67aa1ff0ce4446e29a69a31754191f9b P eb5d098367f54935b85164679a41e8bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:30:58.216509 2626 raft_consensus.cc:2258] T 67aa1ff0ce4446e29a69a31754191f9b P a487e45250c24103ae92207d5e7a05de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:30:58.218132 2626 ts_tablet_manager.cc:1905] T 67aa1ff0ce4446e29a69a31754191f9b P a487e45250c24103ae92207d5e7a05de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:30:58.218215 2628 ts_tablet_manager.cc:1905] T 67aa1ff0ce4446e29a69a31754191f9b P eb5d098367f54935b85164679a41e8bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:30:58.220605 2628 ts_tablet_manager.cc:1918] T 67aa1ff0ce4446e29a69a31754191f9b P eb5d098367f54935b85164679a41e8bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:30:58.220726 2628 log.cc:1196] T 67aa1ff0ce4446e29a69a31754191f9b P eb5d098367f54935b85164679a41e8bc: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/wal/wals/67aa1ff0ce4446e29a69a31754191f9b
I20260929 10:30:58.221029 2628 ts_tablet_manager.cc:1939] T 67aa1ff0ce4446e29a69a31754191f9b P eb5d098367f54935b85164679a41e8bc: Deleting consensus metadata
I20260929 10:30:58.221601 2626 ts_tablet_manager.cc:1918] T 67aa1ff0ce4446e29a69a31754191f9b P a487e45250c24103ae92207d5e7a05de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:30:58.221716 2626 log.cc:1196] T 67aa1ff0ce4446e29a69a31754191f9b P a487e45250c24103ae92207d5e7a05de: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/wal/wals/67aa1ff0ce4446e29a69a31754191f9b
I20260929 10:30:58.222069 2626 ts_tablet_manager.cc:1939] T 67aa1ff0ce4446e29a69a31754191f9b P a487e45250c24103ae92207d5e7a05de: Deleting consensus metadata
I20260929 10:30:58.222343 2113 catalog_manager.cc:4841] TS eb5d098367f54935b85164679a41e8bc (127.20.92.3:33341): tablet 67aa1ff0ce4446e29a69a31754191f9b (table test-table [id=c419c4b49e914163a6c95c154a6f03e6]) successfully deleted
I20260929 10:30:58.223232 2112 catalog_manager.cc:4841] TS a487e45250c24103ae92207d5e7a05de (127.20.92.2:42265): tablet 67aa1ff0ce4446e29a69a31754191f9b (table test-table [id=c419c4b49e914163a6c95c154a6f03e6]) successfully deleted
W20260929 10:30:58.223670 2112 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.92.1:39955 (error 108)
I20260929 10:30:58.223918 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 2322
I20260929 10:30:58.225531 2598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:58.225176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:39955 (local address 127.0.0.1:58536)
0929 10:30:58.225345 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:58.225426 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.1:39955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 10:30:58.225788 2112 catalog_manager.cc:4584] TS f224cceb75ea443f9e00bac46dc83f00 (127.20.92.1:39955): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 67aa1ff0ce4446e29a69a31754191f9b: Network error: Client connection negotiation failed: client connection to 127.20.92.1:39955: connect: Connection refused (error 111)
I20260929 10:30:58.229683 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 2456
I20260929 10:30:58.235296 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 2094
I20260929 10:30:58.242039 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:45535
--webserver_interface=127.20.92.62
--webserver_port=37651
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:42149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:45535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:30:58.381028 2630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:30:58.381356 2630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:30:58.381624 2630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:30:58.387120 2630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:30:58.387233 2630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:30:58.387259 2630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:30:58.387275 2630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:30:58.387291 2630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:30:58.394589 2630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:42149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:45535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:45535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=37651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:30:58.395740 2630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:30:58.396742 2630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:30:58.410785 2637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:58.410785 2639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:58.410816 2636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:58.411109 2630 server_base.cc:1034] running on GCE node
I20260929 10:30:58.411826 2630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:30:58.413036 2630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:30:58.414232 2630 hybrid_clock.cc:648] HybridClock initialized: now 1790677858414194 us; error 51 us; skew 500 ppm
Sep 29 10:30:58 dist-test-slave-9cj5 krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677858, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:30:58.417265 2630 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:30:58.418320 2630 webserver.cc:460] Webserver started at http://127.20.92.62:37651/ using document root <none> and password file <none>
I20260929 10:30:58.418946 2630 fs_manager.cc:359] Metadata directory not provided
I20260929 10:30:58.419050 2630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:30:58.423236 2630 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:30:58.425618 2646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:58.427049 2630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:30:58.427211 2630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "6ba76be930574b13a58cb1c54c7bfb17"
format_stamp: "Formatted at 2026-09-29 10:30:57 on dist-test-slave-9cj5"
server_key: "b3e58075d162ef95de99a4a377591629"
server_key_iv: "d1d1505ff10e3d5c2b7cd954e8611c3d"
server_key_version: "encryptionkey@0"
I20260929 10:30:58.427922 2630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:30:58.441576 2630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:30:58.444774 2630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:30:58.444994 2630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:30:58.452458 2630 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:45535
I20260929 10:30:58.452867 2698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:45535 every 8 connection(s)
I20260929 10:30:58.453649 2630 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:30:58.458247 2699 sys_catalog.cc:263] Verifying existing consensus state
I20260929 10:30:58.459975 2699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17: Bootstrap starting.
I20260929 10:30:58.459997 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 2630
I20260929 10:30:58.460510 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:39955
--local_ip_for_outbound_sockets=127.20.92.1
--tserver_master_addrs=127.20.92.62:45535
--webserver_port=42723
--webserver_interface=127.20.92.1
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:42149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:30:58.477942 2699 log.cc:824] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17: Log is configured to *not* fsync() on all Append() calls
I20260929 10:30:58.487365 2699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:30:58.487788 2699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17: Bootstrap complete.
I20260929 10:30:58.492020 2699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba76be930574b13a58cb1c54c7bfb17" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45535 } }
I20260929 10:30:58.492417 2699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ba76be930574b13a58cb1c54c7bfb17, State: Initialized, Role: FOLLOWER
I20260929 10:30:58.492919 2699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba76be930574b13a58cb1c54c7bfb17" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45535 } }
I20260929 10:30:58.493038 2699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:30:58.493086 2699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:30:58.493358 2699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [term 1 FOLLOWER]: Advancing to term 2
I20260929 10:30:58.495425 2699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba76be930574b13a58cb1c54c7bfb17" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45535 } }
I20260929 10:30:58.495777 2699 leader_election.cc:302] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ba76be930574b13a58cb1c54c7bfb17; no voters:
I20260929 10:30:58.496163 2699 leader_election.cc:288] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 10:30:58.496253 2703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 10:30:58.496456 2703 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [term 2 LEADER]: Becoming Leader. State: Replica: 6ba76be930574b13a58cb1c54c7bfb17, State: Running, Role: LEADER
I20260929 10:30:58.496752 2703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba76be930574b13a58cb1c54c7bfb17" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45535 } }
I20260929 10:30:58.497473 2699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:30:58.497723 2705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ba76be930574b13a58cb1c54c7bfb17. Latest consensus state: current_term: 2 leader_uuid: "6ba76be930574b13a58cb1c54c7bfb17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba76be930574b13a58cb1c54c7bfb17" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45535 } } }
I20260929 10:30:58.497870 2705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [sys.catalog]: This master's current role is: LEADER
I20260929 10:30:58.498176 2704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6ba76be930574b13a58cb1c54c7bfb17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba76be930574b13a58cb1c54c7bfb17" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45535 } } }
I20260929 10:30:58.498282 2704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [sys.catalog]: This master's current role is: LEADER
I20260929 10:30:58.498620 2707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:30:58.503556 2707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:30:58.504290 2707 catalog_manager.cc:1259] Loaded cluster ID: 18bf720afb1045a98ff4c386f9c130e9
I20260929 10:30:58.504347 2707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:30:58.505782 2707 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 10:30:58.510771 2707 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 10:30:58.516569 2707 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:30:58.517421 2707 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17: Loaded TSK: 0
I20260929 10:30:58.518680 2707 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 10:30:58.582046 2701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:30:58.582329 2701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:30:58.582399 2701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:30:58.586591 2701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:30:58.586678 2701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:30:58.586771 2701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:30:58.592080 2701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:42149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:39955
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=42723
--tserver_master_addrs=127.20.92.62:45535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:30:58.593084 2701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:30:58.594121 2701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:30:58.604295 2729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:58.604296 2726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:58.604296 2727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:58.604858 2701 server_base.cc:1034] running on GCE node
I20260929 10:30:58.605329 2701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:30:58.606061 2701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:30:58.607270 2701 hybrid_clock.cc:648] HybridClock initialized: now 1790677858607249 us; error 40 us; skew 500 ppm
Sep 29 10:30:58 dist-test-slave-9cj5 krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677858, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:30:58.616309 2701 init.cc:377] Logged in from keytab as kudu/127.20.92.1@KRBTEST.COM (short username kudu)
I20260929 10:30:58.617501 2701 webserver.cc:460] Webserver started at http://127.20.92.1:42723/ using document root <none> and password file <none>
I20260929 10:30:58.618155 2701 fs_manager.cc:359] Metadata directory not provided
I20260929 10:30:58.618232 2701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:30:58.622403 2701 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:30:58.624716 2736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:58.626178 2701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:30:58.626323 2701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "f224cceb75ea443f9e00bac46dc83f00"
format_stamp: "Formatted at 2026-09-29 10:30:57 on dist-test-slave-9cj5"
server_key: "4a39cd1d4b8f2536fbc59dee176b280c"
server_key_iv: "1af9113c9b11d67036be856f5e211e22"
server_key_version: "encryptionkey@0"
I20260929 10:30:58.626928 2701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:30:58.641582 2701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:30:58.644681 2701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:30:58.644891 2701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:30:58.645498 2701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:30:58.646705 2743 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 10:30:58.650899 2701 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 10:30:58.650964 2701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260929 10:30:58.651036 2701 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 10:30:58.652662 2701 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 10:30:58.652721 2701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 10:30:58.652968 2743 tablet_bootstrap.cc:492] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00: Bootstrap starting.
I20260929 10:30:58.663743 2701 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:39955
I20260929 10:30:58.663869 2850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:39955 every 8 connection(s)
I20260929 10:30:58.665027 2701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
I20260929 10:30:58.667698 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 2701
I20260929 10:30:58.668262 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:42265
--local_ip_for_outbound_sockets=127.20.92.2
--tserver_master_addrs=127.20.92.62:45535
--webserver_port=41047
--webserver_interface=127.20.92.2
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:42149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:30:58.673727 2743 log.cc:824] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00: Log is configured to *not* fsync() on all Append() calls
Sep 29 10:30:58 dist-test-slave-9cj5 krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677858, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:30:58.683701 2743 tablet_bootstrap.cc:492] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:30:58.684186 2743 tablet_bootstrap.cc:492] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00: Bootstrap complete.
I20260929 10:30:58.685273 2854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:58.667255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.1:60763)
0929 10:30:58.668252 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:58.668312 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:58.669903 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:58.671087 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:58.671095 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:58.671532 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:30:58.672004 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:58.672020 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.673357 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.673360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:58.674296 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:58.674306 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.675374 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.675377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:58.675564 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:58.676320 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:30:58.676340 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:30:58.678461 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:30:58.681141 (+ 2680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:58.681149 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:58.681175 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:58.681814 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:58.682429 (+ 615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:58.682435 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:58.682439 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:58.682672 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:58.683432 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:30:58.683439 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:30:58.683743 (+ 304us) client_negotiation.cc:769] Sending connection context
0929 10:30:58.683937 (+ 194us) client_negotiation.cc:240] Negotiation successful
0929 10:30:58.684285 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":200,"threads_started":1}
I20260929 10:30:58.685577 2743 ts_tablet_manager.cc:1397] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00: Time spent bootstrapping tablet: real 0.033s user 0.020s sys 0.007s
I20260929 10:30:58.685828 2853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:58.667194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:60763 (local address 127.20.92.62:45535)
0929 10:30:58.667800 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:58.667825 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:58.668710 (+ 885us) server_negotiation.cc:407] Connection header received
0929 10:30:58.670355 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:58.670366 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:58.670677 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:58.671028 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:58.672203 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.673176 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.674426 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.675127 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.675369 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:58.678705 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:58.678712 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:58.678728 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:58.678763 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:58.680992 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:58.682078 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:58.682091 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:58.682097 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:58.682288 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:58.682833 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:58.682837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:58.682839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:58.683299 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:58.683441 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:58.684097 (+ 656us) server_negotiation.cc:299] Negotiation successful
0929 10:30:58.684481 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":145,"threads_started":1}
I20260929 10:30:58.688513 2851 heartbeater.cc:347] Connected to a master server at 127.20.92.62:45535
I20260929 10:30:58.688755 2851 heartbeater.cc:464] Registering TS with master...
I20260929 10:30:58.688411 2743 raft_consensus.cc:348] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a487e45250c24103ae92207d5e7a05de" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 42265 } } peers { permanent_uuid: "f224cceb75ea443f9e00bac46dc83f00" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39955 } } peers { permanent_uuid: "eb5d098367f54935b85164679a41e8bc" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 33341 } }
I20260929 10:30:58.688982 2743 raft_consensus.cc:729] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f224cceb75ea443f9e00bac46dc83f00, State: Initialized, Role: FOLLOWER
I20260929 10:30:58.689322 2851 heartbeater.cc:511] Master 127.20.92.62:45535 requested a full tablet report, sending...
I20260929 10:30:58.689522 2743 consensus_queue.cc:260] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a487e45250c24103ae92207d5e7a05de" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 42265 } } peers { permanent_uuid: "f224cceb75ea443f9e00bac46dc83f00" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39955 } } peers { permanent_uuid: "eb5d098367f54935b85164679a41e8bc" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 33341 } }
I20260929 10:30:58.690925 2743 ts_tablet_manager.cc:1428] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260929 10:30:58.692198 2663 ts_manager.cc:194] Registered new tserver with Master: f224cceb75ea443f9e00bac46dc83f00 (127.20.92.1:39955)
I20260929 10:30:58.695765 2663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.1@KRBTEST.COM'} at 127.20.92.1:60763
I20260929 10:30:58.699184 2851 heartbeater.cc:503] Master 127.20.92.62:45535 was elected leader, sending a full tablet report...
Sep 29 10:30:58 dist-test-slave-9cj5 krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677858, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for kudu/127.20.92.1@KRBTEST.COM
I20260929 10:30:58.706789 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:58.695092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:39955 (local address 127.0.0.1:58542)
0929 10:30:58.695887 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:58.695934 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:58.696087 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:58.696901 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:58.696914 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:58.697397 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:30:58.697526 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:58.697534 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.698921 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.698925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:58.699878 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:58.699890 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.700791 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.700794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:58.700893 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:58.701450 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:30:58.701461 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:30:58.703047 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:30:58.704974 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:58.704983 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:58.704988 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:58.705218 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:58.705522 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:58.705526 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:58.705528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:58.705619 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:58.706128 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:30:58.706134 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:30:58.706262 (+ 128us) client_negotiation.cc:769] Sending connection context
0929 10:30:58.706404 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 10:30:58.706599 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":155,"threads_started":1}
I20260929 10:30:58.706789 2859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:58.695224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58542 (local address 127.20.92.1:39955)
0929 10:30:58.695613 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:58.695625 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:58.696044 (+ 419us) server_negotiation.cc:407] Connection header received
0929 10:30:58.696247 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:58.696252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:58.696453 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:58.696599 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:58.697663 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.698758 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.700005 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.700545 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.700663 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:58.703196 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:58.703206 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:58.703218 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:58.703249 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:58.704742 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:58.705336 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:58.705342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:58.705344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:58.705426 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:58.705717 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:58.705722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:58.705723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:58.705996 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:58.706161 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:58.706481 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 10:30:58.706638 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":297,"thread_start_us":108,"threads_started":1}
I20260929 10:30:58.707739 2785 tablet_service.cc:1514] Processing DeleteTablet for tablet 67aa1ff0ce4446e29a69a31754191f9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:30:58 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:58542
I20260929 10:30:58.708592 2860 tablet_replica.cc:330] stopping tablet replica
I20260929 10:30:58.709007 2860 raft_consensus.cc:2229] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:30:58.709288 2860 raft_consensus.cc:2258] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:30:58.710456 2860 ts_tablet_manager.cc:1905] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:30:58.715191 2860 ts_tablet_manager.cc:1918] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:30:58.715296 2860 log.cc:1196] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/wal/wals/67aa1ff0ce4446e29a69a31754191f9b
I20260929 10:30:58.715591 2860 ts_tablet_manager.cc:1939] T 67aa1ff0ce4446e29a69a31754191f9b P f224cceb75ea443f9e00bac46dc83f00: Deleting consensus metadata
I20260929 10:30:58.717597 2649 catalog_manager.cc:4841] TS f224cceb75ea443f9e00bac46dc83f00 (127.20.92.1:39955): tablet 67aa1ff0ce4446e29a69a31754191f9b (table test-table [id=c419c4b49e914163a6c95c154a6f03e6]) successfully deleted
W20260929 10:30:58.799113 2855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:30:58.799561 2855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:30:58.799645 2855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:30:58.805677 2855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:30:58.805792 2855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:30:58.805904 2855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:30:58.814383 2855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:42149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:42265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=41047
--tserver_master_addrs=127.20.92.62:45535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:30:58.815501 2855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:30:58.816552 2855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:30:58.826186 2865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:58.831948 2866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:58.831948 2868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:58.832331 2855 server_base.cc:1034] running on GCE node
I20260929 10:30:58.832741 2855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:30:58.833346 2855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:30:58.834506 2855 hybrid_clock.cc:648] HybridClock initialized: now 1790677858834453 us; error 65 us; skew 500 ppm
Sep 29 10:30:58 dist-test-slave-9cj5 krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677858, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:30:58.837137 2855 init.cc:377] Logged in from keytab as kudu/127.20.92.2@KRBTEST.COM (short username kudu)
I20260929 10:30:58.838038 2855 webserver.cc:460] Webserver started at http://127.20.92.2:41047/ using document root <none> and password file <none>
I20260929 10:30:58.838680 2855 fs_manager.cc:359] Metadata directory not provided
I20260929 10:30:58.838783 2855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:30:58.842703 2855 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 10:30:58.844802 2875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:58.846148 2855 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 10:30:58.846279 2855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "a487e45250c24103ae92207d5e7a05de"
format_stamp: "Formatted at 2026-09-29 10:30:57 on dist-test-slave-9cj5"
server_key: "fdf95d4cf95f9c3896a02a1d615a6fd6"
server_key_iv: "616a4df8942a7fde84035efcc2e4c73a"
server_key_version: "encryptionkey@0"
I20260929 10:30:58.846836 2855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:30:58.864619 2855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:30:58.867717 2855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:30:58.867928 2855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:30:58.868533 2855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:30:58.869414 2855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:30:58.869488 2855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:58.869558 2855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:30:58.869604 2855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:58.878763 2855 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:42265
I20260929 10:30:58.878830 2988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:42265 every 8 connection(s)
I20260929 10:30:58.879762 2855 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
Sep 29 10:30:58 dist-test-slave-9cj5 krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677858, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:30:58.889321 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 2855
I20260929 10:30:58.889737 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:33341
--local_ip_for_outbound_sockets=127.20.92.3
--tserver_master_addrs=127.20.92.62:45535
--webserver_port=34939
--webserver_interface=127.20.92.3
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:42149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:30:58.894675 2853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:58.881710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:58223 (local address 127.20.92.62:45535)
0929 10:30:58.881906 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:58.881912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:58.882685 (+ 773us) server_negotiation.cc:407] Connection header received
0929 10:30:58.884033 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:58.884036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:58.884087 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:58.884166 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:58.885331 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.885892 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.887040 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.887290 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.887381 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:58.890116 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:58.890119 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:58.890121 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:58.890150 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:58.892077 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:58.892635 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:58.892642 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:58.892647 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:58.892793 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:58.893136 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:58.893142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:58.893147 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:58.893467 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:58.893588 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:58.894251 (+ 663us) server_negotiation.cc:299] Negotiation successful
0929 10:30:58.894430 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 10:30:58.895418 2991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:58.882063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.2:58223)
0929 10:30:58.882521 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:58.882563 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:58.883828 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:58.884332 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:58.884338 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:58.884685 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:30:58.885184 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:58.885197 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.886001 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.886004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:58.886912 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:58.886921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:58.887426 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:58.887429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:58.887566 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:58.888156 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:30:58.888172 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:30:58.889907 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:30:58.892226 (+ 2319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:58.892231 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:58.892245 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:58.892519 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:58.892885 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:58.892888 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:58.892889 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:58.893039 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:58.893630 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:30:58.893642 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:30:58.893978 (+ 336us) client_negotiation.cc:769] Sending connection context
0929 10:30:58.894156 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 10:30:58.894439 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":109,"threads_started":1}
I20260929 10:30:58.896703 2989 heartbeater.cc:347] Connected to a master server at 127.20.92.62:45535
I20260929 10:30:58.896934 2989 heartbeater.cc:464] Registering TS with master...
I20260929 10:30:58.897495 2989 heartbeater.cc:511] Master 127.20.92.62:45535 requested a full tablet report, sending...
I20260929 10:30:58.898679 2663 ts_manager.cc:194] Registered new tserver with Master: a487e45250c24103ae92207d5e7a05de (127.20.92.2:42265)
I20260929 10:30:58.899314 2663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.2@KRBTEST.COM'} at 127.20.92.2:58223
W20260929 10:30:59.007709 2992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:30:59.007970 2992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:30:59.008015 2992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:30:59.012141 2992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:30:59.012209 2992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:30:59.012307 2992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:30:59.017795 2992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:42149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:33341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=34939
--tserver_master_addrs=127.20.92.62:45535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:30:59.018855 2992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:30:59.019977 2992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:30:59.029896 2997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:59.029923 2998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:59.029910 3000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:59.030055 2992 server_base.cc:1034] running on GCE node
I20260929 10:30:59.030704 2992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:30:59.031493 2992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:30:59.032691 2992 hybrid_clock.cc:648] HybridClock initialized: now 1790677859032680 us; error 35 us; skew 500 ppm
Sep 29 10:30:59 dist-test-slave-9cj5 krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677859, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:30:59.035459 2992 init.cc:377] Logged in from keytab as kudu/127.20.92.3@KRBTEST.COM (short username kudu)
I20260929 10:30:59.036355 2992 webserver.cc:460] Webserver started at http://127.20.92.3:34939/ using document root <none> and password file <none>
I20260929 10:30:59.036929 2992 fs_manager.cc:359] Metadata directory not provided
I20260929 10:30:59.037039 2992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:30:59.040906 2992 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:30:59.043068 3007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:59.044348 2992 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:30:59.044445 2992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "eb5d098367f54935b85164679a41e8bc"
format_stamp: "Formatted at 2026-09-29 10:30:58 on dist-test-slave-9cj5"
server_key: "5193a2dfda4eff069a12b3b348e7bd8c"
server_key_iv: "6b2244dd640c4e719dc1bb8ace354704"
server_key_version: "encryptionkey@0"
I20260929 10:30:59.044947 2992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:30:59.054792 2992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:30:59.057876 2992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:30:59.058027 2992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:30:59.058554 2992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:30:59.059307 2992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:30:59.059353 2992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:59.059391 2992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:30:59.059406 2992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:59.067843 2992 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:33341
I20260929 10:30:59.067862 3120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:33341 every 8 connection(s)
I20260929 10:30:59.068813 2992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:30:59.077993 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 2992
Sep 29 10:30:59 dist-test-slave-9cj5 krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677859, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:30:59.082715 2853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:59.070604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:33693 (local address 127.20.92.62:45535)
0929 10:30:59.070768 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:59.070773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:59.071615 (+ 842us) server_negotiation.cc:407] Connection header received
0929 10:30:59.072901 (+ 1286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:59.072903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:59.072948 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:59.073022 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:59.074159 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:59.074670 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:59.075763 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:59.075968 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:59.076042 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:59.078853 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:59.078858 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:59.078861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:59.078891 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:59.080591 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:59.081094 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:59.081098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:59.081100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:59.081184 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:59.081535 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:59.081538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:59.081540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:59.081727 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:59.081870 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:59.082421 (+ 551us) server_negotiation.cc:299] Negotiation successful
0929 10:30:59.082537 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 10:30:59.083498 3123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:59.071028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.3:33693)
0929 10:30:59.071463 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:59.071502 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:59.072708 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:59.073189 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:59.073195 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:59.073538 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:30:59.074033 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:59.074045 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:59.074776 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:59.074779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:59.075658 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:59.075667 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:59.076129 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:59.076134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:59.076333 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:59.076861 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:30:59.076877 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:30:59.078689 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:30:59.080709 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:59.080715 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:59.080730 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:59.080988 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:59.081282 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:30:59.081285 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:30:59.081287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:30:59.081430 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:30:59.081868 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:30:59.081875 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:30:59.082141 (+ 266us) client_negotiation.cc:769] Sending connection context
0929 10:30:59.082366 (+ 225us) client_negotiation.cc:240] Negotiation successful
0929 10:30:59.082686 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":117,"threads_started":1}
I20260929 10:30:59.084707 3121 heartbeater.cc:347] Connected to a master server at 127.20.92.62:45535
I20260929 10:30:59.084962 3121 heartbeater.cc:464] Registering TS with master...
I20260929 10:30:59.085449 3121 heartbeater.cc:511] Master 127.20.92.62:45535 requested a full tablet report, sending...
I20260929 10:30:59.086537 2663 ts_manager.cc:194] Registered new tserver with Master: eb5d098367f54935b85164679a41e8bc (127.20.92.3:33341)
I20260929 10:30:59.087107 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:59.078701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41658 (local address 127.20.92.62:45535)
0929 10:30:59.079025 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:59.079028 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:59.079136 (+ 108us) server_negotiation.cc:407] Connection header received
0929 10:30:59.079299 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:59.079303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:59.079357 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:59.079453 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:30:59.080426 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:59.080968 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:59.081913 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:59.082172 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:59.082267 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:59.083771 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:30:59.083775 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:30:59.083777 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:30:59.083808 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:30:59.085337 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:59.085852 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:59.085856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:59.085857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:59.085938 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:30:59.086217 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:30:59.086221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:30:59.086223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:30:59.086393 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:30:59.086600 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:59.086760 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 10:30:59.086942 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":263,"thread_start_us":92,"threads_started":1}
I20260929 10:30:59.087302 2663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.3@KRBTEST.COM'} at 127.20.92.3:33693
I20260929 10:30:59.088896 20848 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:30:59.093267 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:59.089490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41664 (local address 127.20.92.62:45535)
0929 10:30:59.089692 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:59.089696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:59.089890 (+ 194us) server_negotiation.cc:407] Connection header received
0929 10:30:59.090092 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:59.090094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:59.090175 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:59.090280 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:30:59.090529 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:59.091023 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:59.091890 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:59.092125 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:59.092239 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:59.092551 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:30:59.092900 (+ 349us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:30:59.093038 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:59.093082 (+ 44us) server_negotiation.cc:299] Negotiation successful
0929 10:30:59.093135 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260929 10:30:59.093973 2663 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:30:59.097215 2663 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:30:59.105209 2785 tablet_service.cc:1467] Processing CreateTablet for tablet 8313115a9a364ef697e40a2d86db833e (DEFAULT_TABLE table=test-table [id=e2e719ec2fca40c9b646e316eeb991c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:30:59.105957 2785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8313115a9a364ef697e40a2d86db833e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:30:59.108848 2743 tablet_bootstrap.cc:492] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00: Bootstrap starting.
I20260929 10:30:59.109633 3130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:59.104340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:33341 (local address 127.0.0.1:52620)
0929 10:30:59.104639 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:59.104654 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:59.104733 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:59.105524 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:59.105527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:59.105550 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:30:59.105663 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:59.105668 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:59.106858 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:59.106863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:59.108130 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:59.108141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:59.109170 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:59.109177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:59.109281 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:59.109324 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 10:30:59.109373 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 10:30:59.109445 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":110,"thread_start_us":97,"threads_started":1}
I20260929 10:30:59.109799 2743 tablet_bootstrap.cc:654] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00: Neither blocks nor log segments found. Creating new log.
I20260929 10:30:59.109835 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:59.104031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:42265 (local address 127.0.0.1:49194)
0929 10:30:59.104187 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:59.104202 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:59.104280 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:59.105449 (+ 1169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:59.105452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:59.105468 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:30:59.105592 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:59.105600 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:59.107097 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:59.107101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:59.107963 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:59.107972 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:59.109301 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:59.109305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:59.109405 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:59.109430 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 10:30:59.109521 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 10:30:59.109597 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":26}
I20260929 10:30:59.110170 3132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:59.104485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49194 (local address 127.20.92.2:42265)
0929 10:30:59.104972 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:59.104996 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:59.105019 (+ 23us) server_negotiation.cc:407] Connection header received
0929 10:30:59.105091 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:59.105096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:59.105279 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:59.105779 (+ 500us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:30:59.105937 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:59.106954 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:59.108104 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:59.109147 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:59.109275 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:59.109433 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:59.109556 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 10:30:59.109652 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":141,"threads_started":1}
I20260929 10:30:59.110479 3131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:59.104515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52620 (local address 127.20.92.3:33341)
0929 10:30:59.104984 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:59.104993 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:59.105013 (+ 20us) server_negotiation.cc:407] Connection header received
0929 10:30:59.105078 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:59.105081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:59.105247 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:59.105410 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:30:59.105802 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:59.106733 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:59.108252 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:59.109024 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:59.109975 (+ 951us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:59.110120 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:59.110171 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 10:30:59.110297 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":110,"threads_started":1}
I20260929 10:30:59.111519 2743 tablet_bootstrap.cc:492] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00: No bootstrap required, opened a new log
I20260929 10:30:59.111624 2743 ts_tablet_manager.cc:1397] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 10:30:59.112058 2743 raft_consensus.cc:348] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a487e45250c24103ae92207d5e7a05de" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 42265 } } peers { permanent_uuid: "eb5d098367f54935b85164679a41e8bc" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 33341 } } peers { permanent_uuid: "f224cceb75ea443f9e00bac46dc83f00" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39955 } }
I20260929 10:30:59.112183 2743 raft_consensus.cc:374] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:30:59.112222 2743 raft_consensus.cc:729] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f224cceb75ea443f9e00bac46dc83f00, State: Initialized, Role: FOLLOWER
I20260929 10:30:59.112433 2743 consensus_queue.cc:260] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a487e45250c24103ae92207d5e7a05de" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 42265 } } peers { permanent_uuid: "eb5d098367f54935b85164679a41e8bc" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 33341 } } peers { permanent_uuid: "f224cceb75ea443f9e00bac46dc83f00" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39955 } }
I20260929 10:30:59.112788 2743 ts_tablet_manager.cc:1428] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 10:30:59.113464 2923 tablet_service.cc:1467] Processing CreateTablet for tablet 8313115a9a364ef697e40a2d86db833e (DEFAULT_TABLE table=test-table [id=e2e719ec2fca40c9b646e316eeb991c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:30:59.113467 3055 tablet_service.cc:1467] Processing CreateTablet for tablet 8313115a9a364ef697e40a2d86db833e (DEFAULT_TABLE table=test-table [id=e2e719ec2fca40c9b646e316eeb991c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:30:59.114542 3055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8313115a9a364ef697e40a2d86db833e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:30:59.114535 2923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8313115a9a364ef697e40a2d86db833e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:30:59.121471 3134 tablet_bootstrap.cc:492] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc: Bootstrap starting.
I20260929 10:30:59.121481 3135 tablet_bootstrap.cc:492] T 8313115a9a364ef697e40a2d86db833e P a487e45250c24103ae92207d5e7a05de: Bootstrap starting.
I20260929 10:30:59.123502 3134 tablet_bootstrap.cc:654] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc: Neither blocks nor log segments found. Creating new log.
I20260929 10:30:59.123899 3135 tablet_bootstrap.cc:654] T 8313115a9a364ef697e40a2d86db833e P a487e45250c24103ae92207d5e7a05de: Neither blocks nor log segments found. Creating new log.
I20260929 10:30:59.124812 3134 log.cc:824] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc: Log is configured to *not* fsync() on all Append() calls
I20260929 10:30:59.124854 3135 log.cc:824] T 8313115a9a364ef697e40a2d86db833e P a487e45250c24103ae92207d5e7a05de: Log is configured to *not* fsync() on all Append() calls
I20260929 10:30:59.126581 3135 tablet_bootstrap.cc:492] T 8313115a9a364ef697e40a2d86db833e P a487e45250c24103ae92207d5e7a05de: No bootstrap required, opened a new log
I20260929 10:30:59.126652 3134 tablet_bootstrap.cc:492] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc: No bootstrap required, opened a new log
I20260929 10:30:59.126816 3135 ts_tablet_manager.cc:1397] T 8313115a9a364ef697e40a2d86db833e P a487e45250c24103ae92207d5e7a05de: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 10:30:59.126847 3134 ts_tablet_manager.cc:1397] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 10:30:59.129539 3135 raft_consensus.cc:348] T 8313115a9a364ef697e40a2d86db833e P a487e45250c24103ae92207d5e7a05de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a487e45250c24103ae92207d5e7a05de" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 42265 } } peers { permanent_uuid: "eb5d098367f54935b85164679a41e8bc" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 33341 } } peers { permanent_uuid: "f224cceb75ea443f9e00bac46dc83f00" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39955 } }
I20260929 10:30:59.129580 3134 raft_consensus.cc:348] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a487e45250c24103ae92207d5e7a05de" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 42265 } } peers { permanent_uuid: "eb5d098367f54935b85164679a41e8bc" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 33341 } } peers { permanent_uuid: "f224cceb75ea443f9e00bac46dc83f00" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39955 } }
I20260929 10:30:59.129823 3135 raft_consensus.cc:374] T 8313115a9a364ef697e40a2d86db833e P a487e45250c24103ae92207d5e7a05de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:30:59.129863 3134 raft_consensus.cc:374] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:30:59.129884 3135 raft_consensus.cc:729] T 8313115a9a364ef697e40a2d86db833e P a487e45250c24103ae92207d5e7a05de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a487e45250c24103ae92207d5e7a05de, State: Initialized, Role: FOLLOWER
I20260929 10:30:59.129925 3134 raft_consensus.cc:729] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb5d098367f54935b85164679a41e8bc, State: Initialized, Role: FOLLOWER
I20260929 10:30:59.130426 3135 consensus_queue.cc:260] T 8313115a9a364ef697e40a2d86db833e P a487e45250c24103ae92207d5e7a05de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a487e45250c24103ae92207d5e7a05de" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 42265 } } peers { permanent_uuid: "eb5d098367f54935b85164679a41e8bc" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 33341 } } peers { permanent_uuid: "f224cceb75ea443f9e00bac46dc83f00" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39955 } }
I20260929 10:30:59.130470 3134 consensus_queue.cc:260] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a487e45250c24103ae92207d5e7a05de" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 42265 } } peers { permanent_uuid: "eb5d098367f54935b85164679a41e8bc" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 33341 } } peers { permanent_uuid: "f224cceb75ea443f9e00bac46dc83f00" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39955 } }
I20260929 10:30:59.131072 2989 heartbeater.cc:503] Master 127.20.92.62:45535 was elected leader, sending a full tablet report...
I20260929 10:30:59.131095 3121 heartbeater.cc:503] Master 127.20.92.62:45535 was elected leader, sending a full tablet report...
I20260929 10:30:59.131639 3134 ts_tablet_manager.cc:1428] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 10:30:59.132369 3135 ts_tablet_manager.cc:1428] T 8313115a9a364ef697e40a2d86db833e P a487e45250c24103ae92207d5e7a05de: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260929 10:30:59.166308 2852 tablet.cc:2369] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:30:59.215925 3139 raft_consensus.cc:482] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:30:59.216173 3139 raft_consensus.cc:504] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a487e45250c24103ae92207d5e7a05de" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 42265 } } peers { permanent_uuid: "eb5d098367f54935b85164679a41e8bc" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 33341 } } peers { permanent_uuid: "f224cceb75ea443f9e00bac46dc83f00" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39955 } }
I20260929 10:30:59.217338 3139 leader_election.cc:288] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a487e45250c24103ae92207d5e7a05de (127.20.92.2:42265), f224cceb75ea443f9e00bac46dc83f00 (127.20.92.1:39955)
I20260929 10:30:59.222362 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:59.218102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:43687 (local address 127.20.92.1:39955)
0929 10:30:59.218360 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:59.218364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:59.218407 (+ 43us) server_negotiation.cc:407] Connection header received
0929 10:30:59.218532 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:59.218535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:59.218619 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:59.218760 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:30:59.218976 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:59.219666 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:59.221008 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:59.221672 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:59.221787 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:59.221937 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:59.222152 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 10:30:59.222230 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":87,"threads_started":1}
I20260929 10:30:59.222361 3132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:59.217833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:56341 (local address 127.20.92.2:42265)
0929 10:30:59.217984 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:59.217989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:59.218036 (+ 47us) server_negotiation.cc:407] Connection header received
0929 10:30:59.218095 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:59.218098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:59.218147 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:59.218249 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:30:59.218624 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:59.219412 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:59.220984 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:59.221583 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:59.221699 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:59.221800 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:59.222164 (+ 364us) server_negotiation.cc:299] Negotiation successful
0929 10:30:59.222233 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 10:30:59.222431 3123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:59.217682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:42265 (local address 127.20.92.3:56341)
0929 10:30:59.217834 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:59.217851 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:59.217946 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:59.218331 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:59.218336 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:59.218360 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:30:59.218496 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:59.218503 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:59.219521 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:59.219524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:59.220835 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:59.220846 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:59.221707 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:59.221710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:59.221905 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:59.222037 (+ 132us) client_negotiation.cc:769] Sending connection context
0929 10:30:59.222120 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 10:30:59.222259 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":101}
I20260929 10:30:59.222445 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:59.217984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:39955 (local address 127.20.92.3:43687)
0929 10:30:59.218331 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:59.218343 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:59.218433 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:59.218772 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:30:59.218774 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:30:59.218788 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:30:59.218869 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:59.218873 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:59.219776 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:59.219779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:59.220898 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:30:59.220907 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:30:59.221796 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:30:59.221799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:30:59.221959 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:59.222043 (+ 84us) client_negotiation.cc:769] Sending connection context
0929 10:30:59.222120 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 10:30:59.222259 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":105,"thread_start_us":154,"threads_started":1}
I20260929 10:30:59.223114 2943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8313115a9a364ef697e40a2d86db833e" candidate_uuid: "eb5d098367f54935b85164679a41e8bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a487e45250c24103ae92207d5e7a05de" is_pre_election: true
I20260929 10:30:59.223099 2805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8313115a9a364ef697e40a2d86db833e" candidate_uuid: "eb5d098367f54935b85164679a41e8bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f224cceb75ea443f9e00bac46dc83f00" is_pre_election: true
I20260929 10:30:59.223413 2805 raft_consensus.cc:2454] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb5d098367f54935b85164679a41e8bc in term 0.
I20260929 10:30:59.223420 2943 raft_consensus.cc:2454] T 8313115a9a364ef697e40a2d86db833e P a487e45250c24103ae92207d5e7a05de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb5d098367f54935b85164679a41e8bc in term 0.
I20260929 10:30:59.223872 3010 leader_election.cc:302] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eb5d098367f54935b85164679a41e8bc, f224cceb75ea443f9e00bac46dc83f00; no voters:
I20260929 10:30:59.224136 3139 raft_consensus.cc:2781] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:30:59.224241 3139 raft_consensus.cc:482] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:30:59.224323 3139 raft_consensus.cc:3037] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:30:59.225354 3139 raft_consensus.cc:504] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a487e45250c24103ae92207d5e7a05de" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 42265 } } peers { permanent_uuid: "eb5d098367f54935b85164679a41e8bc" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 33341 } } peers { permanent_uuid: "f224cceb75ea443f9e00bac46dc83f00" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39955 } }
I20260929 10:30:59.225699 3139 leader_election.cc:288] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc [CANDIDATE]: Term 1 election: Requested vote from peers a487e45250c24103ae92207d5e7a05de (127.20.92.2:42265), f224cceb75ea443f9e00bac46dc83f00 (127.20.92.1:39955)
I20260929 10:30:59.226168 2805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8313115a9a364ef697e40a2d86db833e" candidate_uuid: "eb5d098367f54935b85164679a41e8bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f224cceb75ea443f9e00bac46dc83f00"
I20260929 10:30:59.226171 2943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8313115a9a364ef697e40a2d86db833e" candidate_uuid: "eb5d098367f54935b85164679a41e8bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a487e45250c24103ae92207d5e7a05de"
I20260929 10:30:59.226279 2943 raft_consensus.cc:3037] T 8313115a9a364ef697e40a2d86db833e P a487e45250c24103ae92207d5e7a05de [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:30:59.226305 2805 raft_consensus.cc:3037] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:30:59.227238 2805 raft_consensus.cc:2454] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb5d098367f54935b85164679a41e8bc in term 1.
I20260929 10:30:59.227396 2943 raft_consensus.cc:2454] T 8313115a9a364ef697e40a2d86db833e P a487e45250c24103ae92207d5e7a05de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb5d098367f54935b85164679a41e8bc in term 1.
I20260929 10:30:59.227536 3010 leader_election.cc:302] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eb5d098367f54935b85164679a41e8bc, f224cceb75ea443f9e00bac46dc83f00; no voters:
I20260929 10:30:59.227849 3139 raft_consensus.cc:2781] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:30:59.228089 3139 raft_consensus.cc:686] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc [term 1 LEADER]: Becoming Leader. State: Replica: eb5d098367f54935b85164679a41e8bc, State: Running, Role: LEADER
I20260929 10:30:59.228425 3139 consensus_queue.cc:237] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a487e45250c24103ae92207d5e7a05de" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 42265 } } peers { permanent_uuid: "eb5d098367f54935b85164679a41e8bc" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 33341 } } peers { permanent_uuid: "f224cceb75ea443f9e00bac46dc83f00" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39955 } }
I20260929 10:30:59.231652 2663 catalog_manager.cc:5495] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc reported cstate change: term changed from 0 to 1, leader changed from <none> to eb5d098367f54935b85164679a41e8bc (127.20.92.3). New cstate: current_term: 1 leader_uuid: "eb5d098367f54935b85164679a41e8bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a487e45250c24103ae92207d5e7a05de" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 42265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb5d098367f54935b85164679a41e8bc" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 33341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f224cceb75ea443f9e00bac46dc83f00" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39955 } health_report { overall_health: UNKNOWN } } }
I20260929 10:30:59.267020 3131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:59.263400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52636 (local address 127.20.92.3:33341)
0929 10:30:59.263535 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 10:30:59.263539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:30:59.263551 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:30:59.263624 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:30:59.263626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:30:59.263686 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:30:59.263776 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:30:59.263980 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:59.264483 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:59.265363 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:30:59.265605 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:30:59.265696 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:30:59.266101 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:30:59.266521 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:30:59.266612 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 10:30:59.266752 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 10:30:59.266803 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 10:30:59.272747 2805 raft_consensus.cc:1261] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [term 1 FOLLOWER]: Refusing update from remote peer eb5d098367f54935b85164679a41e8bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:30:59.272778 2943 raft_consensus.cc:1261] T 8313115a9a364ef697e40a2d86db833e P a487e45250c24103ae92207d5e7a05de [term 1 FOLLOWER]: Refusing update from remote peer eb5d098367f54935b85164679a41e8bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:30:59.273434 3139 consensus_queue.cc:1035] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f224cceb75ea443f9e00bac46dc83f00" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:30:59.273614 3142 consensus_queue.cc:1035] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a487e45250c24103ae92207d5e7a05de" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 42265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:30:59.280251 3146 mvcc.cc:205] Tried to move back new op lower bound from 7334616511574208512 to 7334616511402446848. Current Snapshot: MvccSnapshot[applied={T|T < 7334616511574208512}]
I20260929 10:30:59.289578 2662 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:30:59.289832 2662 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:30:59.292281 2662 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17: Sending DeleteTablet for 3 replicas of tablet 8313115a9a364ef697e40a2d86db833e
I20260929 10:30:59.293071 3055 tablet_service.cc:1514] Processing DeleteTablet for tablet 8313115a9a364ef697e40a2d86db833e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:30:59 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:52620
I20260929 10:30:59.293273 2785 tablet_service.cc:1514] Processing DeleteTablet for tablet 8313115a9a364ef697e40a2d86db833e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:30:59 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:58542
I20260929 10:30:59.293372 2923 tablet_service.cc:1514] Processing DeleteTablet for tablet 8313115a9a364ef697e40a2d86db833e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:30:59 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:49194
I20260929 10:30:59.293672 3156 tablet_replica.cc:330] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00: stopping tablet replica
I20260929 10:30:59.293843 3156 raft_consensus.cc:2229] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:30:59.293883 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 2701
I20260929 10:30:59.293889 3157 tablet_replica.cc:330] T 8313115a9a364ef697e40a2d86db833e P a487e45250c24103ae92207d5e7a05de: stopping tablet replica
I20260929 10:30:59.294883 3157 raft_consensus.cc:2229] T 8313115a9a364ef697e40a2d86db833e P a487e45250c24103ae92207d5e7a05de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:30:59.294994 3155 tablet_replica.cc:330] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc: stopping tablet replica
I20260929 10:30:59.295251 3155 raft_consensus.cc:2229] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:30:59.295365 3157 raft_consensus.cc:2258] T 8313115a9a364ef697e40a2d86db833e P a487e45250c24103ae92207d5e7a05de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:30:59.295647 3155 raft_consensus.cc:2258] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:30:59.297211 3157 ts_tablet_manager.cc:1905] T 8313115a9a364ef697e40a2d86db833e P a487e45250c24103ae92207d5e7a05de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:30:59.299201 3155 ts_tablet_manager.cc:1905] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:30:59.300139 3157 ts_tablet_manager.cc:1918] T 8313115a9a364ef697e40a2d86db833e P a487e45250c24103ae92207d5e7a05de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:30:59.300275 3157 log.cc:1196] T 8313115a9a364ef697e40a2d86db833e P a487e45250c24103ae92207d5e7a05de: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/wal/wals/8313115a9a364ef697e40a2d86db833e
I20260929 10:30:59.300623 3157 ts_tablet_manager.cc:1939] T 8313115a9a364ef697e40a2d86db833e P a487e45250c24103ae92207d5e7a05de: Deleting consensus metadata
W20260929 10:30:59.300959 2649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.92.1:39955 (error 108)
I20260929 10:30:59.302106 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 2855
I20260929 10:30:59.302233 2649 catalog_manager.cc:4841] TS a487e45250c24103ae92207d5e7a05de (127.20.92.2:42265): tablet 8313115a9a364ef697e40a2d86db833e (table test-table [id=e2e719ec2fca40c9b646e316eeb991c7]) successfully deleted
I20260929 10:30:59.303741 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:59.303399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:39955 (local address 127.0.0.1:58558)
0929 10:30:59.303557 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:59.303640 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.1:39955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 10:30:59.303896 2649 catalog_manager.cc:4584] TS f224cceb75ea443f9e00bac46dc83f00 (127.20.92.1:39955): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8313115a9a364ef697e40a2d86db833e: Network error: Client connection negotiation failed: client connection to 127.20.92.1:39955: connect: Connection refused (error 111)
I20260929 10:30:59.307265 3155 ts_tablet_manager.cc:1918] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:30:59.307358 3155 log.cc:1196] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/wal/wals/8313115a9a364ef697e40a2d86db833e
I20260929 10:30:59.307677 3155 ts_tablet_manager.cc:1939] T 8313115a9a364ef697e40a2d86db833e P eb5d098367f54935b85164679a41e8bc: Deleting consensus metadata
I20260929 10:30:59.308805 2650 catalog_manager.cc:4841] TS eb5d098367f54935b85164679a41e8bc (127.20.92.3:33341): tablet 8313115a9a364ef697e40a2d86db833e (table test-table [id=e2e719ec2fca40c9b646e316eeb991c7]) successfully deleted
I20260929 10:30:59.309245 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 2992
I20260929 10:30:59.315045 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 2630
I20260929 10:30:59.321223 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:45535
--webserver_interface=127.20.92.62
--webserver_port=37651
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:42149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:45535
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:30:59.430936 3159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:30:59.431221 3159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:30:59.431263 3159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:30:59.435245 3159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:30:59.435302 3159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:30:59.435322 3159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:30:59.435339 3159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:30:59.435354 3159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:30:59.440586 3159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:42149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:45535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:45535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=37651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:30:59.441583 3159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:30:59.442528 3159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:30:59.451078 3165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:59.451082 3167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:59.451082 3164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:59.451278 3159 server_base.cc:1034] running on GCE node
I20260929 10:30:59.452056 3159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:30:59.453245 3159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:30:59.454447 3159 hybrid_clock.cc:648] HybridClock initialized: now 1790677859454432 us; error 29 us; skew 500 ppm
Sep 29 10:30:59 dist-test-slave-9cj5 krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677859, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:30:59.457358 3159 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:30:59.458357 3159 webserver.cc:460] Webserver started at http://127.20.92.62:37651/ using document root <none> and password file <none>
I20260929 10:30:59.458913 3159 fs_manager.cc:359] Metadata directory not provided
I20260929 10:30:59.459004 3159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:30:59.462852 3159 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 10:30:59.464998 3174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:59.466493 3159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:30:59.466655 3159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "6ba76be930574b13a58cb1c54c7bfb17"
format_stamp: "Formatted at 2026-09-29 10:30:57 on dist-test-slave-9cj5"
server_key: "b3e58075d162ef95de99a4a377591629"
server_key_iv: "d1d1505ff10e3d5c2b7cd954e8611c3d"
server_key_version: "encryptionkey@0"
I20260929 10:30:59.467208 3159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:30:59.477475 3159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:30:59.480829 3159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:30:59.481045 3159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:30:59.488363 3159 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:45535
I20260929 10:30:59.488659 3226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:45535 every 8 connection(s)
I20260929 10:30:59.489552 3159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:30:59.493835 3227 sys_catalog.cc:263] Verifying existing consensus state
I20260929 10:30:59.495399 3227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17: Bootstrap starting.
I20260929 10:30:59.497673 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 3159
I20260929 10:30:59.498224 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:39955
--local_ip_for_outbound_sockets=127.20.92.1
--tserver_master_addrs=127.20.92.62:45535
--webserver_port=42723
--webserver_interface=127.20.92.1
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:42149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:30:59.513399 3227 log.cc:824] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17: Log is configured to *not* fsync() on all Append() calls
I20260929 10:30:59.526297 3227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:30:59.526733 3227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17: Bootstrap complete.
I20260929 10:30:59.531001 3227 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba76be930574b13a58cb1c54c7bfb17" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45535 } }
I20260929 10:30:59.531390 3227 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ba76be930574b13a58cb1c54c7bfb17, State: Initialized, Role: FOLLOWER
I20260929 10:30:59.531886 3227 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba76be930574b13a58cb1c54c7bfb17" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45535 } }
I20260929 10:30:59.531999 3227 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:30:59.532085 3227 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:30:59.532200 3227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [term 2 FOLLOWER]: Advancing to term 3
I20260929 10:30:59.534252 3227 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba76be930574b13a58cb1c54c7bfb17" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45535 } }
I20260929 10:30:59.534590 3227 leader_election.cc:302] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ba76be930574b13a58cb1c54c7bfb17; no voters:
I20260929 10:30:59.534925 3227 leader_election.cc:288] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 10:30:59.535022 3231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 10:30:59.535214 3231 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [term 3 LEADER]: Becoming Leader. State: Replica: 6ba76be930574b13a58cb1c54c7bfb17, State: Running, Role: LEADER
I20260929 10:30:59.535506 3231 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba76be930574b13a58cb1c54c7bfb17" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45535 } }
I20260929 10:30:59.536186 3227 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:30:59.536463 3231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ba76be930574b13a58cb1c54c7bfb17. Latest consensus state: current_term: 3 leader_uuid: "6ba76be930574b13a58cb1c54c7bfb17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba76be930574b13a58cb1c54c7bfb17" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45535 } } }
I20260929 10:30:59.536581 3231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [sys.catalog]: This master's current role is: LEADER
I20260929 10:30:59.536921 3235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:30:59.536948 3232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6ba76be930574b13a58cb1c54c7bfb17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba76be930574b13a58cb1c54c7bfb17" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45535 } } }
I20260929 10:30:59.537179 3232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ba76be930574b13a58cb1c54c7bfb17 [sys.catalog]: This master's current role is: LEADER
I20260929 10:30:59.542232 3235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:30:59.543042 3235 catalog_manager.cc:1259] Loaded cluster ID: 18bf720afb1045a98ff4c386f9c130e9
I20260929 10:30:59.543121 3235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 10:30:59.543916 3235 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f986bf479ad google::LogMessage::Fail() at ??:0
@ 0x7f986bf4a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f986bf474de google::LogMessage::Flush() at ??:0
@ 0x7f986bf48a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9876886fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f9876887d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9876884579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f98768c8640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f986e1993d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f986e199c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f986e19b3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f986e18a512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f986f7ce6db start_thread at ??:0
@ 0x7f986aa4561f clone at ??:0
W20260929 10:30:59.648180 3229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:30:59.648552 3229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:30:59.648628 3229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:30:59.654923 3229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:30:59.655014 3229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:30:59.655103 3229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:30:59.663368 3229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:42149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:39955
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=42723
--tserver_master_addrs=127.20.92.62:45535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:30:59.664821 3229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:30:59.666173 3229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:30:59.677631 3268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:59.677621 3269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:59.677621 3271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:59.677994 3229 server_base.cc:1034] running on GCE node
I20260929 10:30:59.678516 3229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:30:59.679231 3229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:30:59.680431 3229 hybrid_clock.cc:648] HybridClock initialized: now 1790677859680406 us; error 44 us; skew 500 ppm
Sep 29 10:30:59 dist-test-slave-9cj5 krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677859, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:30:59.683231 3229 init.cc:377] Logged in from keytab as kudu/127.20.92.1@KRBTEST.COM (short username kudu)
I20260929 10:30:59.684178 3229 webserver.cc:460] Webserver started at http://127.20.92.1:42723/ using document root <none> and password file <none>
I20260929 10:30:59.684739 3229 fs_manager.cc:359] Metadata directory not provided
I20260929 10:30:59.684800 3229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:30:59.688886 3229 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:30:59.691407 3278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:59.692767 3229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 10:30:59.692938 3229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "f224cceb75ea443f9e00bac46dc83f00"
format_stamp: "Formatted at 2026-09-29 10:30:57 on dist-test-slave-9cj5"
server_key: "4a39cd1d4b8f2536fbc59dee176b280c"
server_key_iv: "1af9113c9b11d67036be856f5e211e22"
server_key_version: "encryptionkey@0"
I20260929 10:30:59.693535 3229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:30:59.721251 3229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:30:59.725490 3229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:30:59.725873 3229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:30:59.726504 3229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:30:59.728078 3285 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 10:30:59.733999 3229 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 10:30:59.734118 3229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260929 10:30:59.734261 3229 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 10:30:59.736003 3229 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 10:30:59.736073 3229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 10:30:59.736400 3285 tablet_bootstrap.cc:492] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00: Bootstrap starting.
I20260929 10:30:59.746845 3229 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:39955
I20260929 10:30:59.746872 3392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:39955 every 8 connection(s)
I20260929 10:30:59.747865 3229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
I20260929 10:30:59.756323 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 3229
I20260929 10:30:59.756800 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:42265
--local_ip_for_outbound_sockets=127.20.92.2
--tserver_master_addrs=127.20.92.62:45535
--webserver_port=41047
--webserver_interface=127.20.92.2
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:42149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:30:59.760164 3285 log.cc:824] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00: Log is configured to *not* fsync() on all Append() calls
I20260929 10:30:59.769986 3285 tablet_bootstrap.cc:492] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:30:59.770474 3285 tablet_bootstrap.cc:492] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00: Bootstrap complete.
I20260929 10:30:59.771862 3285 ts_tablet_manager.cc:1397] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00: Time spent bootstrapping tablet: real 0.036s user 0.029s sys 0.004s
I20260929 10:30:59.775445 3285 raft_consensus.cc:348] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a487e45250c24103ae92207d5e7a05de" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 42265 } } peers { permanent_uuid: "eb5d098367f54935b85164679a41e8bc" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 33341 } } peers { permanent_uuid: "f224cceb75ea443f9e00bac46dc83f00" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39955 } }
I20260929 10:30:59.776003 3285 raft_consensus.cc:729] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f224cceb75ea443f9e00bac46dc83f00, State: Initialized, Role: FOLLOWER
I20260929 10:30:59.776543 3285 consensus_queue.cc:260] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a487e45250c24103ae92207d5e7a05de" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 42265 } } peers { permanent_uuid: "eb5d098367f54935b85164679a41e8bc" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 33341 } } peers { permanent_uuid: "f224cceb75ea443f9e00bac46dc83f00" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39955 } }
I20260929 10:30:59.777647 3285 ts_tablet_manager.cc:1428] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
W20260929 10:30:59.789148 3395 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:30:59.749942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.1:52161)
0929 10:30:59.750440 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:59.750493 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 10:30:59.751321 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:30:59.788343 (+ 37022us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":156,"threads_started":1}
W20260929 10:30:59.789686 3393 heartbeater.cc:644] Failed to heartbeat to 127.20.92.62:45535 (0 consecutive failures): Network error: Failed to ping master at 127.20.92.62:45535: Client connection negotiation failed: client connection to 127.20.92.62:45535: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 10:30:59.790505 3395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:59.790267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.1:49319)
0929 10:30:59.790390 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:59.790452 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 10:30:59.791251 3395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:59.791064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.1:44993)
0929 10:30:59.791164 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:59.791204 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 10:30:59.791479 3393 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 10:30:59.878252 3396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:30:59.878525 3396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:30:59.878611 3396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:30:59.882927 3396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:30:59.882997 3396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:30:59.883083 3396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:30:59.888790 3396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:42149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:42265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=41047
--tserver_master_addrs=127.20.92.62:45535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:30:59.889950 3396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:30:59.891000 3396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:30:59.901259 3404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:59.901259 3406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:30:59.901547 3403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:30:59.901760 3396 server_base.cc:1034] running on GCE node
I20260929 10:30:59.902202 3396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:30:59.902930 3396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:30:59.904127 3396 hybrid_clock.cc:648] HybridClock initialized: now 1790677859904113 us; error 34 us; skew 500 ppm
Sep 29 10:30:59 dist-test-slave-9cj5 krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677859, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:30:59.907137 3396 init.cc:377] Logged in from keytab as kudu/127.20.92.2@KRBTEST.COM (short username kudu)
I20260929 10:30:59.908025 3396 webserver.cc:460] Webserver started at http://127.20.92.2:41047/ using document root <none> and password file <none>
I20260929 10:30:59.908633 3396 fs_manager.cc:359] Metadata directory not provided
I20260929 10:30:59.908726 3396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:30:59.912843 3396 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:30:59.915356 3413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:59.916797 3396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 10:30:59.916983 3396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "a487e45250c24103ae92207d5e7a05de"
format_stamp: "Formatted at 2026-09-29 10:30:57 on dist-test-slave-9cj5"
server_key: "fdf95d4cf95f9c3896a02a1d615a6fd6"
server_key_iv: "616a4df8942a7fde84035efcc2e4c73a"
server_key_version: "encryptionkey@0"
I20260929 10:30:59.917582 3396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:30:59.946085 3396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:30:59.949155 3396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:30:59.949371 3396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:30:59.950001 3396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:30:59.950827 3396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:30:59.950924 3396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:59.951018 3396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:30:59.951063 3396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:30:59.960147 3396 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:42265
I20260929 10:30:59.960199 3526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:42265 every 8 connection(s)
I20260929 10:30:59.961145 3396 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
I20260929 10:30:59.964154 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 3396
I20260929 10:30:59.964633 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:33341
--local_ip_for_outbound_sockets=127.20.92.3
--tserver_master_addrs=127.20.92.62:45535
--webserver_port=34939
--webserver_interface=127.20.92.3
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:42149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:30:59.965732 3529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:59.963410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.2:46973)
0929 10:30:59.963938 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:59.964105 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":118,"threads_started":1}
W20260929 10:30:59.966382 3527 heartbeater.cc:644] Failed to heartbeat to 127.20.92.62:45535 (0 consecutive failures): Network error: Failed to ping master at 127.20.92.62:45535: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
I20260929 10:30:59.967171 3529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:59.966904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.2:36187)
0929 10:30:59.967040 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:59.967095 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 10:30:59.967875 3529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:30:59.967637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.2:37923)
0929 10:30:59.967777 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 10:30:59.967822 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 10:30:59.968061 3527 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 10:31:00.118321 3530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:31:00.118618 3530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:31:00.118664 3530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:31:00.122889 3530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:31:00.122956 3530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:31:00.123018 3530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:31:00.128427 3530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:42149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:33341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=34939
--tserver_master_addrs=127.20.92.62:45535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:31:00.129483 3530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:31:00.130550 3530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:31:00.140355 3535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:31:00.140355 3538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:31:00.140395 3536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:31:00.140702 3530 server_base.cc:1034] running on GCE node
I20260929 10:31:00.141232 3530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:31:00.141929 3530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:31:00.143137 3530 hybrid_clock.cc:648] HybridClock initialized: now 1790677860143118 us; error 28 us; skew 500 ppm
Sep 29 10:31:00 dist-test-slave-9cj5 krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677860, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:31:00.146067 3530 init.cc:377] Logged in from keytab as kudu/127.20.92.3@KRBTEST.COM (short username kudu)
I20260929 10:31:00.147097 3530 webserver.cc:460] Webserver started at http://127.20.92.3:34939/ using document root <none> and password file <none>
I20260929 10:31:00.147723 3530 fs_manager.cc:359] Metadata directory not provided
I20260929 10:31:00.147822 3530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:31:00.152073 3530 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 10:31:00.154423 3545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:31:00.156232 3530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 10:31:00.156409 3530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "eb5d098367f54935b85164679a41e8bc"
format_stamp: "Formatted at 2026-09-29 10:30:58 on dist-test-slave-9cj5"
server_key: "5193a2dfda4eff069a12b3b348e7bd8c"
server_key_iv: "6b2244dd640c4e719dc1bb8ace354704"
server_key_version: "encryptionkey@0"
I20260929 10:31:00.157008 3530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:31:00.190155 3530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:31:00.193348 3530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:31:00.193581 3530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:31:00.194307 3530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:31:00.195196 3530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:31:00.195271 3530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:31:00.195343 3530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:31:00.195376 3530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:31:00.204856 3530 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:33341
I20260929 10:31:00.204878 3658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:33341 every 8 connection(s)
I20260929 10:31:00.205928 3530 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:31:00.209981 3661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:00.208166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.3:45477)
0929 10:31:00.208631 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:00.208745 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":109,"threads_started":1}
W20260929 10:31:00.210405 3659 heartbeater.cc:644] Failed to heartbeat to 127.20.92.62:45535 (0 consecutive failures): Network error: Failed to ping master at 127.20.92.62:45535: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
I20260929 10:31:00.211026 3661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:00.210803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.3:53007)
0929 10:31:00.210920 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:00.210975 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 10:31:00.211608 3661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:00.211408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.3:55271)
0929 10:31:00.211513 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:00.211556 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 10:31:00.211799 3659 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 10:31:00.214274 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 3530
I20260929 10:31:00.792469 3663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:00.792042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.1:35113)
0929 10:31:00.792300 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:00.792357 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":100,"threads_started":1}
I20260929 10:31:00.969048 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:00.968642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.2:40423)
0929 10:31:00.968894 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:00.968949 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":85,"threads_started":1}
I20260929 10:31:01.212929 3665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:01.212408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.3:36557)
0929 10:31:01.212730 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:01.212791 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":74,"threads_started":1}
I20260929 10:31:01.394310 3666 raft_consensus.cc:482] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:31:01.394583 3666 raft_consensus.cc:504] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a487e45250c24103ae92207d5e7a05de" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 42265 } } peers { permanent_uuid: "eb5d098367f54935b85164679a41e8bc" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 33341 } } peers { permanent_uuid: "f224cceb75ea443f9e00bac46dc83f00" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39955 } }
I20260929 10:31:01.395884 3666 leader_election.cc:288] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a487e45250c24103ae92207d5e7a05de (127.20.92.2:42265), eb5d098367f54935b85164679a41e8bc (127.20.92.3:33341)
Sep 29 10:31:01 dist-test-slave-9cj5 krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677859, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for kudu/127.20.92.2@KRBTEST.COM
Sep 29 10:31:01 dist-test-slave-9cj5 krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677859, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for kudu/127.20.92.3@KRBTEST.COM
I20260929 10:31:01.414017 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:01.396161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:33341 (local address 127.20.92.1:33883)
0929 10:31:01.396510 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:01.396526 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:31:01.396615 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:31:01.398596 (+ 1981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:31:01.398603 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:31:01.399274 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:31:01.400185 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:01.400207 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:01.402192 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:01.402195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:01.403793 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:01.403805 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:01.404827 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:01.404831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:01.405140 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:01.406162 (+ 1022us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:31:01.406186 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:31:01.408320 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:31:01.410750 (+ 2430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:31:01.410758 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:31:01.410773 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:31:01.411266 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:31:01.411754 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:31:01.411757 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:31:01.411759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:31:01.411978 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:31:01.412808 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:31:01.412815 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:31:01.413186 (+ 371us) client_negotiation.cc:769] Sending connection context
0929 10:31:01.413407 (+ 221us) client_negotiation.cc:240] Negotiation successful
0929 10:31:01.413848 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":708,"spinlock_wait_cycles":37632,"thread_start_us":65,"threads_started":1}
I20260929 10:31:01.414017 3668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:01.396046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:42265 (local address 127.20.92.1:36899)
0929 10:31:01.396401 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:01.396416 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:31:01.396529 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:31:01.398596 (+ 2067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:31:01.398603 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:31:01.399274 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:31:01.400185 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:01.400207 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:01.402192 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:01.402195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:01.403795 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:01.403805 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:01.404827 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:01.404831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:01.405135 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:01.406139 (+ 1004us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:31:01.406185 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:31:01.408025 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:31:01.410750 (+ 2725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:31:01.410758 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:31:01.410773 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:31:01.411266 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:31:01.411748 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:31:01.411752 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:31:01.411754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:31:01.411978 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:31:01.412808 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:31:01.412815 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:31:01.413186 (+ 371us) client_negotiation.cc:769] Sending connection context
0929 10:31:01.413407 (+ 221us) client_negotiation.cc:240] Negotiation successful
0929 10:31:01.413797 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":334,"spinlock_wait_cycles":22912,"thread_start_us":84,"threads_started":1}
I20260929 10:31:01.414089 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:01.396369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:33883 (local address 127.20.92.3:33341)
0929 10:31:01.396779 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0929 10:31:01.396797 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 10:31:01.396826 (+ 29us) server_negotiation.cc:407] Connection header received
0929 10:31:01.397628 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:31:01.397637 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:31:01.398122 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:31:01.398449 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:31:01.400401 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:01.402013 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:01.403945 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:01.404681 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:01.404900 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:01.408520 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:31:01.408529 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:31:01.408547 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:31:01.408583 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:31:01.410612 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:31:01.411417 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:31:01.411424 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:31:01.411430 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:31:01.411658 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:31:01.412087 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:31:01.412091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:31:01.412092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:31:01.412690 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:31:01.412803 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 10:31:01.413603 (+ 800us) server_negotiation.cc:299] Negotiation successful
0929 10:31:01.413893 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":116,"threads_started":1}
I20260929 10:31:01.414733 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:01.396215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:36899 (local address 127.20.92.2:42265)
0929 10:31:01.396627 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0929 10:31:01.396654 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 10:31:01.396698 (+ 44us) server_negotiation.cc:407] Connection header received
0929 10:31:01.397628 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:31:01.397637 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:31:01.398122 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:31:01.398449 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:31:01.400401 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:01.402027 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:01.403945 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:01.404703 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:01.404900 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:01.408327 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:31:01.408332 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:31:01.408345 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:31:01.408403 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:31:01.410612 (+ 2209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:31:01.411417 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:31:01.411424 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:31:01.411430 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:31:01.411614 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:31:01.412087 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:31:01.412100 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:31:01.412102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:31:01.412670 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:31:01.412803 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 10:31:01.413553 (+ 750us) server_negotiation.cc:299] Negotiation successful
0929 10:31:01.413893 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":65,"threads_started":1}
W20260929 10:31:01.417053 3282 leader_election.cc:341] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb5d098367f54935b85164679a41e8bc (127.20.92.3:33341): Not found: Tablet not found: 8313115a9a364ef697e40a2d86db833e
W20260929 10:31:01.417300 3281 leader_election.cc:341] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a487e45250c24103ae92207d5e7a05de (127.20.92.2:42265): Not found: Tablet not found: 8313115a9a364ef697e40a2d86db833e
I20260929 10:31:01.417379 3281 leader_election.cc:302] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f224cceb75ea443f9e00bac46dc83f00; no voters: a487e45250c24103ae92207d5e7a05de, eb5d098367f54935b85164679a41e8bc
I20260929 10:31:01.417707 3666 raft_consensus.cc:2726] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 10:31:01.793974 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:01.793535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.1:53503)
0929 10:31:01.793739 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:01.793885 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260929 10:31:01.970765 3671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:01.970191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.2:58393)
0929 10:31:01.970582 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:01.970659 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":138,"threads_started":1}
I20260929 10:31:02.214336 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:02.213874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.3:59593)
0929 10:31:02.214172 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:02.214235 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":78,"threads_started":1}
I20260929 10:31:02.795291 3673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:02.794827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.1:48109)
0929 10:31:02.795115 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:02.795184 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":79,"threads_started":1}
I20260929 10:31:02.971979 3674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:02.971562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.2:43925)
0929 10:31:02.971828 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:02.971885 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":91,"threads_started":1}
I20260929 10:31:03.033901 3675 raft_consensus.cc:482] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:31:03.034072 3675 raft_consensus.cc:504] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a487e45250c24103ae92207d5e7a05de" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 42265 } } peers { permanent_uuid: "eb5d098367f54935b85164679a41e8bc" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 33341 } } peers { permanent_uuid: "f224cceb75ea443f9e00bac46dc83f00" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39955 } }
I20260929 10:31:03.034602 3675 leader_election.cc:288] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a487e45250c24103ae92207d5e7a05de (127.20.92.2:42265), eb5d098367f54935b85164679a41e8bc (127.20.92.3:33341)
W20260929 10:31:03.035405 3282 leader_election.cc:341] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb5d098367f54935b85164679a41e8bc (127.20.92.3:33341): Not found: Tablet not found: 8313115a9a364ef697e40a2d86db833e
W20260929 10:31:03.035616 3281 leader_election.cc:341] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a487e45250c24103ae92207d5e7a05de (127.20.92.2:42265): Not found: Tablet not found: 8313115a9a364ef697e40a2d86db833e
I20260929 10:31:03.035706 3281 leader_election.cc:302] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f224cceb75ea443f9e00bac46dc83f00; no voters: a487e45250c24103ae92207d5e7a05de, eb5d098367f54935b85164679a41e8bc
I20260929 10:31:03.035867 3675 raft_consensus.cc:2726] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 10:31:03.215672 3676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:03.215179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.3:42703)
0929 10:31:03.215451 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:03.215509 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":83,"threads_started":1}
I20260929 10:31:03.796677 3677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:03.796215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.1:51573)
0929 10:31:03.796523 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:03.796577 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":90,"threads_started":1}
I20260929 10:31:03.973068 3678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:03.972685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.2:43995)
0929 10:31:03.972925 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:03.972976 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":82,"threads_started":1}
I20260929 10:31:04.217026 3679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:04.216598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.3:39899)
0929 10:31:04.216872 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:04.216932 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":79,"threads_started":1}
I20260929 10:31:04.798159 3680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:04.797676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.1:54229)
0929 10:31:04.798017 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:04.798067 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":71,"threads_started":1}
I20260929 10:31:04.974488 3681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:04.974054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.2:52413)
0929 10:31:04.974325 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:04.974393 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
I20260929 10:31:05.218577 3682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:05.217954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.3:41887)
0929 10:31:05.218343 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:05.218449 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":134,"threads_started":1}
I20260929 10:31:05.799369 3683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:05.798921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.1:45187)
0929 10:31:05.799200 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:05.799260 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":80,"threads_started":1}
I20260929 10:31:05.975682 3684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:05.975213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.2:41439)
0929 10:31:05.975511 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:05.975588 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":71,"threads_started":1}
I20260929 10:31:06.219833 3685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:06.219401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.3:45505)
0929 10:31:06.219675 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:06.219739 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":73,"threads_started":1}
I20260929 10:31:06.316191 3686 raft_consensus.cc:482] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:31:06.316354 3686 raft_consensus.cc:504] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a487e45250c24103ae92207d5e7a05de" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 42265 } } peers { permanent_uuid: "eb5d098367f54935b85164679a41e8bc" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 33341 } } peers { permanent_uuid: "f224cceb75ea443f9e00bac46dc83f00" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39955 } }
I20260929 10:31:06.316784 3686 leader_election.cc:288] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a487e45250c24103ae92207d5e7a05de (127.20.92.2:42265), eb5d098367f54935b85164679a41e8bc (127.20.92.3:33341)
W20260929 10:31:06.317546 3282 leader_election.cc:341] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb5d098367f54935b85164679a41e8bc (127.20.92.3:33341): Not found: Tablet not found: 8313115a9a364ef697e40a2d86db833e
W20260929 10:31:06.317636 3281 leader_election.cc:341] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a487e45250c24103ae92207d5e7a05de (127.20.92.2:42265): Not found: Tablet not found: 8313115a9a364ef697e40a2d86db833e
I20260929 10:31:06.317721 3281 leader_election.cc:302] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f224cceb75ea443f9e00bac46dc83f00; no voters: a487e45250c24103ae92207d5e7a05de, eb5d098367f54935b85164679a41e8bc
I20260929 10:31:06.317879 3686 raft_consensus.cc:2726] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 10:31:06.800724 3687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:06.800237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.1:59239)
0929 10:31:06.800541 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:06.800602 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":100,"threads_started":1}
I20260929 10:31:06.977064 3688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:06.976587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.2:54733)
0929 10:31:06.976897 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:06.976954 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":82,"threads_started":1}
I20260929 10:31:07.221273 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:07.220734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.3:48869)
0929 10:31:07.221021 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:07.221114 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":78,"threads_started":1}
I20260929 10:31:07.802246 3690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:07.801589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.1:54575)
0929 10:31:07.802012 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:07.802097 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":95,"threads_started":1}
I20260929 10:31:07.978303 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:07.977911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.2:50961)
0929 10:31:07.978156 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:07.978210 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":87,"threads_started":1}
I20260929 10:31:08.222576 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:08.222050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.3:35717)
0929 10:31:08.222392 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:08.222453 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":164,"threads_started":1}
I20260929 10:31:08.803406 3693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:08.803026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.1:36897)
0929 10:31:08.803261 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:08.803313 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":77,"threads_started":1}
I20260929 10:31:08.979429 3694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:08.979001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.2:45569)
0929 10:31:08.979253 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:08.979306 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":84,"threads_started":1}
I20260929 10:31:09.223773 3695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:09.223318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.3:56311)
0929 10:31:09.223610 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:09.223675 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":77,"threads_started":1}
I20260929 10:31:09.804874 3696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:09.804330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.1:37569)
0929 10:31:09.804670 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:09.804743 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
I20260929 10:31:09.980904 3697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:09.980358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.2:43975)
0929 10:31:09.980738 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:09.980803 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":106,"threads_started":1}
I20260929 10:31:10.225157 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:10.224713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.3:59939)
0929 10:31:10.225000 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:10.225060 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":93,"threads_started":1}
I20260929 10:31:10.456002 3699 raft_consensus.cc:482] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:31:10.456132 3699 raft_consensus.cc:504] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a487e45250c24103ae92207d5e7a05de" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 42265 } } peers { permanent_uuid: "eb5d098367f54935b85164679a41e8bc" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 33341 } } peers { permanent_uuid: "f224cceb75ea443f9e00bac46dc83f00" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39955 } }
I20260929 10:31:10.456523 3699 leader_election.cc:288] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a487e45250c24103ae92207d5e7a05de (127.20.92.2:42265), eb5d098367f54935b85164679a41e8bc (127.20.92.3:33341)
W20260929 10:31:10.457312 3282 leader_election.cc:341] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb5d098367f54935b85164679a41e8bc (127.20.92.3:33341): Not found: Tablet not found: 8313115a9a364ef697e40a2d86db833e
W20260929 10:31:10.457398 3281 leader_election.cc:341] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a487e45250c24103ae92207d5e7a05de (127.20.92.2:42265): Not found: Tablet not found: 8313115a9a364ef697e40a2d86db833e
I20260929 10:31:10.457516 3281 leader_election.cc:302] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f224cceb75ea443f9e00bac46dc83f00; no voters: a487e45250c24103ae92207d5e7a05de, eb5d098367f54935b85164679a41e8bc
I20260929 10:31:10.457703 3699 raft_consensus.cc:2726] T 8313115a9a364ef697e40a2d86db833e P f224cceb75ea443f9e00bac46dc83f00 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 10:31:10.806249 3700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:10.805717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.1:37673)
0929 10:31:10.806093 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:10.806155 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":83,"threads_started":1}
I20260929 10:31:10.982465 3701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:10.982029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.2:49667)
0929 10:31:10.982307 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:10.982369 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":76,"threads_started":1}
I20260929 10:31:11.226564 3702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:11.226098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.3:43651)
0929 10:31:11.226395 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:11.226458 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":92,"threads_started":1}
I20260929 10:31:11.807601 3703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:11.807006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.1:37303)
0929 10:31:11.807389 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:11.807453 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":92,"threads_started":1}
I20260929 10:31:11.983963 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:11.983444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.2:43533)
0929 10:31:11.983793 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:11.983861 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":84,"threads_started":1}
I20260929 10:31:12.228077 3705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:12.227568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.3:48175)
0929 10:31:12.227907 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:12.227972 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":133,"threads_started":1}
I20260929 10:31:12.809043 3706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:12.808670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.1:35625)
0929 10:31:12.808901 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:12.808952 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
I20260929 10:31:12.985554 3707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:12.985035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.2:57521)
0929 10:31:12.985341 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:12.985399 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":121,"threads_started":1}
I20260929 10:31:13.229661 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:13.229168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.3:52231)
0929 10:31:13.229480 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:13.229547 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":107,"threads_started":1}
I20260929 10:31:13.810451 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:13.809974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.1:35941)
0929 10:31:13.810274 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:13.810350 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":101,"threads_started":1}
I20260929 10:31:13.986977 3710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:13.986474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.2:48579)
0929 10:31:13.986817 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:13.986879 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":102,"threads_started":1}
I20260929 10:31:14.231040 3711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:14.230474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.3:51453)
0929 10:31:14.230848 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:14.230918 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":165,"threads_started":1}
I20260929 10:31:14.812139 3712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:14.811678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.1:33927)
0929 10:31:14.811959 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:14.812023 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":69,"threads_started":1}
I20260929 10:31:14.988598 3713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:14.988086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.2:59179)
0929 10:31:14.988390 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:14.988452 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":87,"threads_started":1}
I20260929 10:31:15.225678 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 3229
I20260929 10:31:15.231519 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 3396
I20260929 10:31:15.233265 3714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:15.232534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45535 (local address 127.20.92.3:45915)
0929 10:31:15.233079 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:15.233130 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":74,"threads_started":1}
I20260929 10:31:15.237331 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 3530
2026-09-29T10:31:15Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19300 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:31:15 dist-test-slave-9cj5 krb5kdc[3717](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:31:15 dist-test-slave-9cj5 krb5kdc[3717](info): set up 2 sockets
Sep 29 10:31:15 dist-test-slave-9cj5 krb5kdc[3717](info): commencing operation
krb5kdc: starting...
W20260929 10:31:16.290889 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:31:16 dist-test-slave-9cj5 krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:31:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:31:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
I20260929 10:31:16.427450 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:41227
--webserver_interface=127.20.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:41227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:31:16.544466 3730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:31:16.544759 3730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:31:16.544800 3730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:31:16.549054 3730 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 10:31:16.549134 3730 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 10:31:16.549150 3730 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 10:31:16.549165 3730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:31:16.549185 3730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:31:16.549203 3730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:31:16.549219 3730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:31:16.549235 3730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:31:16.554932 3730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:41227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:41227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:31:16.555994 3730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:31:16.556958 3730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:31:16.566275 3730 server_base.cc:1034] running on GCE node
W20260929 10:31:16.566221 3735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:31:16.566293 3736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:31:16.566221 3738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:31:16.567123 3730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:31:16.568419 3730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:31:16.569654 3730 hybrid_clock.cc:648] HybridClock initialized: now 1790677876569581 us; error 33 us; skew 500 ppm
Sep 29 10:31:16 dist-test-slave-9cj5 krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677876, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:31:16.572865 3730 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:31:16.574003 3730 webserver.cc:460] Webserver started at http://127.20.92.62:46847/ using document root <none> and password file <none>
I20260929 10:31:16.574582 3730 fs_manager.cc:359] Metadata directory not provided
I20260929 10:31:16.574651 3730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:31:16.574894 3730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:31:16.577000 3730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "f0706ae6bb3b47318e36811afe109005"
format_stamp: "Formatted at 2026-09-29 10:31:16 on dist-test-slave-9cj5"
server_key: "c3b77caac26bdc748e973636cd83034d"
server_key_iv: "eab84ab7f44e19c62da80c319b5dbc23"
server_key_version: "encryptionkey@0"
I20260929 10:31:16.577574 3730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "f0706ae6bb3b47318e36811afe109005"
format_stamp: "Formatted at 2026-09-29 10:31:16 on dist-test-slave-9cj5"
server_key: "c3b77caac26bdc748e973636cd83034d"
server_key_iv: "eab84ab7f44e19c62da80c319b5dbc23"
server_key_version: "encryptionkey@0"
I20260929 10:31:16.581386 3730 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260929 10:31:16.584182 3745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:31:16.585682 3730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 10:31:16.585875 3730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "f0706ae6bb3b47318e36811afe109005"
format_stamp: "Formatted at 2026-09-29 10:31:16 on dist-test-slave-9cj5"
server_key: "c3b77caac26bdc748e973636cd83034d"
server_key_iv: "eab84ab7f44e19c62da80c319b5dbc23"
server_key_version: "encryptionkey@0"
I20260929 10:31:16.586004 3730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:31:16.606736 3730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:31:16.610263 3730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:31:16.610481 3730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:31:16.617923 3730 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:41227
I20260929 10:31:16.618327 3797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:41227 every 8 connection(s)
I20260929 10:31:16.618947 3730 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:31:16.621857 3798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:31:16.624631 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 3730
I20260929 10:31:16.624773 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:31:16.625061 20848 external_mini_cluster.cc:1428] Setting key e99d5680e841f65ea4bd1c1ce7a92967
I20260929 10:31:16.628778 3798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005: Bootstrap starting.
I20260929 10:31:16.631661 3798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005: Neither blocks nor log segments found. Creating new log.
I20260929 10:31:16.632601 3798 log.cc:824] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005: Log is configured to *not* fsync() on all Append() calls
Sep 29 10:31:16 dist-test-slave-9cj5 krb5kdc[3717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:31:16.635063 3798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005: No bootstrap required, opened a new log
I20260929 10:31:16.640501 3798 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0706ae6bb3b47318e36811afe109005" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 41227 } }
I20260929 10:31:16.640914 3798 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:31:16.640976 3798 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0706ae6bb3b47318e36811afe109005, State: Initialized, Role: FOLLOWER
I20260929 10:31:16.641382 3801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:16.626492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59498 (local address 127.20.92.62:41227)
0929 10:31:16.626906 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0929 10:31:16.626922 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 10:31:16.626958 (+ 36us) server_negotiation.cc:407] Connection header received
0929 10:31:16.627611 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:31:16.627627 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:31:16.627963 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:31:16.628239 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:31:16.629251 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:16.630296 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:16.631583 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:16.632135 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:16.632266 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:16.635221 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:31:16.635228 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:31:16.635246 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:31:16.635281 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:31:16.637415 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:31:16.638092 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:31:16.638102 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:31:16.638140 (+ 38us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:31:16.638322 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:31:16.638700 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:31:16.638704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:31:16.638706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:31:16.639306 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:31:16.639527 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 10:31:16.639893 (+ 366us) server_negotiation.cc:299] Negotiation successful
0929 10:31:16.640238 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":136,"threads_started":1}
I20260929 10:31:16.641500 3798 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0706ae6bb3b47318e36811afe109005" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 41227 } }
I20260929 10:31:16.641711 3798 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:31:16.641855 3798 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:31:16.641983 3798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:31:16.643025 3798 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0706ae6bb3b47318e36811afe109005" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 41227 } }
I20260929 10:31:16.643576 3798 leader_election.cc:302] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0706ae6bb3b47318e36811afe109005; no voters:
I20260929 10:31:16.643944 3798 leader_election.cc:288] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:31:16.644145 3803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:31:16.644380 3803 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [term 1 LEADER]: Becoming Leader. State: Replica: f0706ae6bb3b47318e36811afe109005, State: Running, Role: LEADER
I20260929 10:31:16.644743 3803 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0706ae6bb3b47318e36811afe109005" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 41227 } }
I20260929 10:31:16.645346 3798 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:31:16.646539 3804 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0706ae6bb3b47318e36811afe109005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0706ae6bb3b47318e36811afe109005" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 41227 } } }
I20260929 10:31:16.646690 3804 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [sys.catalog]: This master's current role is: LEADER
I20260929 10:31:16.647095 3812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:31:16.647622 3805 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0706ae6bb3b47318e36811afe109005. Latest consensus state: current_term: 1 leader_uuid: "f0706ae6bb3b47318e36811afe109005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0706ae6bb3b47318e36811afe109005" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 41227 } } }
I20260929 10:31:16.647994 3805 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [sys.catalog]: This master's current role is: LEADER
I20260929 10:31:16.650163 3812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:31:16.655412 3812 catalog_manager.cc:1347] Generated new cluster ID: 10bcea7b0f30465b94c67a065b23a4c4
I20260929 10:31:16.655485 3812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:31:16.671769 3812 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:31:16.673136 3812 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:31:16.680188 3812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005: Generated new TSK 0
I20260929 10:31:16.681047 3812 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
I20260929 10:31:16.722572 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:0
--local_ip_for_outbound_sockets=127.20.92.1
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:41227
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:31:16.841969 3824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:31:16.842267 3824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:31:16.842314 3824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:31:16.846738 3824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:31:16.846817 3824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:31:16.846974 3824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:31:16.852555 3824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:41227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:31:16.853645 3824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:31:16.854835 3824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:31:16.865991 3829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:31:16.866005 3830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:31:16.865991 3832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:31:16.867149 3824 server_base.cc:1034] running on GCE node
I20260929 10:31:16.867700 3824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:31:16.868536 3824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:31:16.869735 3824 hybrid_clock.cc:648] HybridClock initialized: now 1790677876869688 us; error 65 us; skew 500 ppm
Sep 29 10:31:16 dist-test-slave-9cj5 krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677876, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:31:16.872905 3824 init.cc:377] Logged in from keytab as kudu/127.20.92.1@KRBTEST.COM (short username kudu)
I20260929 10:31:16.873919 3824 webserver.cc:460] Webserver started at http://127.20.92.1:41693/ using document root <none> and password file <none>
I20260929 10:31:16.874464 3824 fs_manager.cc:359] Metadata directory not provided
I20260929 10:31:16.874522 3824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:31:16.874688 3824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:31:16.877105 3824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/data/instance:
uuid: "842f6d386b2849079c7ce98a3e594a13"
format_stamp: "Formatted at 2026-09-29 10:31:16 on dist-test-slave-9cj5"
server_key: "e5da3ca21baae1b61c8b7a19a93aaf4a"
server_key_iv: "a676b4c99fd4bb9fff7214bdc75124ce"
server_key_version: "encryptionkey@0"
I20260929 10:31:16.877633 3824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance:
uuid: "842f6d386b2849079c7ce98a3e594a13"
format_stamp: "Formatted at 2026-09-29 10:31:16 on dist-test-slave-9cj5"
server_key: "e5da3ca21baae1b61c8b7a19a93aaf4a"
server_key_iv: "a676b4c99fd4bb9fff7214bdc75124ce"
server_key_version: "encryptionkey@0"
I20260929 10:31:16.881994 3824 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 10:31:16.884594 3839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:31:16.886145 3824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:31:16.886299 3824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "842f6d386b2849079c7ce98a3e594a13"
format_stamp: "Formatted at 2026-09-29 10:31:16 on dist-test-slave-9cj5"
server_key: "e5da3ca21baae1b61c8b7a19a93aaf4a"
server_key_iv: "a676b4c99fd4bb9fff7214bdc75124ce"
server_key_version: "encryptionkey@0"
I20260929 10:31:16.886438 3824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:31:16.901578 3824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:31:16.905792 3824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:31:16.906008 3824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:31:16.906719 3824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:31:16.907616 3824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:31:16.907666 3824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:31:16.907709 3824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:31:16.907725 3824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:31:16.917261 3824 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:44141
I20260929 10:31:16.917284 3952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:44141 every 8 connection(s)
I20260929 10:31:16.918272 3824 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
I20260929 10:31:16.920106 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 3824
I20260929 10:31:16.920244 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance
I20260929 10:31:16.920651 20848 external_mini_cluster.cc:1428] Setting key cff016883180cb9c36a1503383108560
Sep 29 10:31:16 dist-test-slave-9cj5 krb5kdc[3717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677876, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.92.2@KRBTEST.COM; defaulting to no policy
I20260929 10:31:16.934798 3801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:16.920542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:37681 (local address 127.20.92.62:41227)
0929 10:31:16.920717 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 10:31:16.920724 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:31:16.921790 (+ 1066us) server_negotiation.cc:407] Connection header received
0929 10:31:16.923271 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:31:16.923274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:31:16.923328 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:31:16.923404 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:31:16.925223 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:16.925915 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:16.927101 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:16.927382 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:16.927506 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:16.930803 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:31:16.930809 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:31:16.930814 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:31:16.930850 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:31:16.932393 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:31:16.933108 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:31:16.933111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:31:16.933113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:31:16.933204 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:31:16.933551 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:31:16.933554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:31:16.933555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:31:16.933841 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:31:16.933925 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 10:31:16.934454 (+ 529us) server_negotiation.cc:299] Negotiation successful
0929 10:31:16.934567 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.2@KRBTEST.COM" created.
I20260929 10:31:16.935726 3955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:16.921066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.1:37681)
0929 10:31:16.921612 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:16.921668 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 10:31:16.923049 (+ 1381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:31:16.923579 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:31:16.923588 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:31:16.924166 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:31:16.924987 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:16.925012 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:16.926048 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:16.926054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:16.926967 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:16.926978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:16.927556 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:16.927560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:16.927808 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:16.928494 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:31:16.928523 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:31:16.930579 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:31:16.932518 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:31:16.932523 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:31:16.932538 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:31:16.932859 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:31:16.933301 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:31:16.933303 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:31:16.933304 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:31:16.933443 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:31:16.933940 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:31:16.933945 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:31:16.934231 (+ 286us) client_negotiation.cc:769] Sending connection context
0929 10:31:16.934398 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 10:31:16.934684 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":129,"threads_started":1}
I20260929 10:31:16.937134 3953 heartbeater.cc:347] Connected to a master server at 127.20.92.62:41227
I20260929 10:31:16.937408 3953 heartbeater.cc:464] Registering TS with master...
I20260929 10:31:16.938009 3953 heartbeater.cc:511] Master 127.20.92.62:41227 requested a full tablet report, sending...
I20260929 10:31:16.939576 3762 ts_manager.cc:194] Registered new tserver with Master: 842f6d386b2849079c7ce98a3e594a13 (127.20.92.1:44141)
I20260929 10:31:16.940732 3762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.1@KRBTEST.COM'} at 127.20.92.1:37681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
I20260929 10:31:16.954684 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:0
--local_ip_for_outbound_sockets=127.20.92.2
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:41227
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:31:17.072271 3958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:31:17.072556 3958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:31:17.072602 3958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:31:17.076942 3958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:31:17.077018 3958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:31:17.077119 3958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:31:17.082988 3958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:41227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:31:17.084136 3958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:31:17.085189 3958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:31:17.095773 3963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:31:17.095763 3964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:31:17.095788 3966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:31:17.095837 3958 server_base.cc:1034] running on GCE node
I20260929 10:31:17.096560 3958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:31:17.097323 3958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:31:17.098529 3958 hybrid_clock.cc:648] HybridClock initialized: now 1790677877098488 us; error 54 us; skew 500 ppm
Sep 29 10:31:17 dist-test-slave-9cj5 krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677877, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:31:17.101940 3958 init.cc:377] Logged in from keytab as kudu/127.20.92.2@KRBTEST.COM (short username kudu)
I20260929 10:31:17.103063 3958 webserver.cc:460] Webserver started at http://127.20.92.2:45625/ using document root <none> and password file <none>
I20260929 10:31:17.103710 3958 fs_manager.cc:359] Metadata directory not provided
I20260929 10:31:17.103811 3958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:31:17.104065 3958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:31:17.106412 3958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/data/instance:
uuid: "f32320fb1e1a4b248864f8b847cadfe9"
format_stamp: "Formatted at 2026-09-29 10:31:17 on dist-test-slave-9cj5"
server_key: "6ba5ddf9ba44f01fd2ef8fc0f6383f79"
server_key_iv: "3901f4ac528d7646f363b96fb09cb649"
server_key_version: "encryptionkey@0"
I20260929 10:31:17.107057 3958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance:
uuid: "f32320fb1e1a4b248864f8b847cadfe9"
format_stamp: "Formatted at 2026-09-29 10:31:17 on dist-test-slave-9cj5"
server_key: "6ba5ddf9ba44f01fd2ef8fc0f6383f79"
server_key_iv: "3901f4ac528d7646f363b96fb09cb649"
server_key_version: "encryptionkey@0"
I20260929 10:31:17.111354 3958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 10:31:17.114411 3973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:31:17.115981 3958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:31:17.116166 3958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "f32320fb1e1a4b248864f8b847cadfe9"
format_stamp: "Formatted at 2026-09-29 10:31:17 on dist-test-slave-9cj5"
server_key: "6ba5ddf9ba44f01fd2ef8fc0f6383f79"
server_key_iv: "3901f4ac528d7646f363b96fb09cb649"
server_key_version: "encryptionkey@0"
I20260929 10:31:17.116312 3958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:31:17.143797 3958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:31:17.148751 3958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:31:17.149003 3958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:31:17.149714 3958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:31:17.150733 3958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:31:17.150812 3958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:31:17.150893 3958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:31:17.150940 3958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:31:17.162618 3958 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:44711
I20260929 10:31:17.162632 4086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:44711 every 8 connection(s)
I20260929 10:31:17.163707 3958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
I20260929 10:31:17.172235 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 3958
I20260929 10:31:17.172387 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance
I20260929 10:31:17.172711 20848 external_mini_cluster.cc:1428] Setting key 418ff7d3906eda35f8c5a5eadc121553
Sep 29 10:31:17 dist-test-slave-9cj5 krb5kdc[3717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677877, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:31:17.182133 3801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:17.166434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:38897 (local address 127.20.92.62:41227)
0929 10:31:17.166623 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 10:31:17.166628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:31:17.167508 (+ 880us) server_negotiation.cc:407] Connection header received
0929 10:31:17.168926 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:31:17.168929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:31:17.168984 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:31:17.169067 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:31:17.170631 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:17.171218 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:17.172422 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:17.172702 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:17.172831 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:17.176649 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:31:17.176653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:31:17.176656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:31:17.176684 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:31:17.178905 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:31:17.179719 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:31:17.179722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:31:17.179724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:31:17.179855 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:31:17.180416 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:31:17.180420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:31:17.180421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:31:17.180722 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:31:17.180815 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 10:31:17.181721 (+ 906us) server_negotiation.cc:299] Negotiation successful
0929 10:31:17.181978 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 10:31:17.183202 4089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:17.166813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.2:38897)
0929 10:31:17.167340 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:17.167386 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 10:31:17.168622 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:31:17.169292 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:31:17.169300 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:31:17.169863 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:31:17.170402 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:17.170416 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:17.171363 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:17.171366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:17.172262 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:17.172272 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:17.172874 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:17.172878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:17.173082 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:17.173842 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:31:17.173861 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:31:17.176155 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:31:17.179156 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:31:17.179163 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:31:17.179190 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:31:17.179597 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:31:17.179977 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:31:17.179984 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:31:17.179989 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:31:17.180245 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:31:17.180865 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:31:17.180876 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:31:17.181299 (+ 423us) client_negotiation.cc:769] Sending connection context
0929 10:31:17.181548 (+ 249us) client_negotiation.cc:240] Negotiation successful
0929 10:31:17.182010 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":136,"threads_started":1}
WARNING: no policy specified for kudu/127.20.92.3@KRBTEST.COM; defaulting to no policy
I20260929 10:31:17.184623 4087 heartbeater.cc:347] Connected to a master server at 127.20.92.62:41227
I20260929 10:31:17.185124 4087 heartbeater.cc:464] Registering TS with master...
I20260929 10:31:17.186107 4087 heartbeater.cc:511] Master 127.20.92.62:41227 requested a full tablet report, sending...
I20260929 10:31:17.187335 3762 ts_manager.cc:194] Registered new tserver with Master: f32320fb1e1a4b248864f8b847cadfe9 (127.20.92.2:44711)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.3@KRBTEST.COM" created.
I20260929 10:31:17.188099 3762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.2@KRBTEST.COM'} at 127.20.92.2:38897
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
I20260929 10:31:17.214078 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:0
--local_ip_for_outbound_sockets=127.20.92.3
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:41227
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:31:17.341823 4092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:31:17.342116 4092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:31:17.342162 4092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:31:17.346585 4092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:31:17.346661 4092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:31:17.346745 4092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:31:17.352831 4092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:41227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:31:17.354110 4092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:31:17.355209 4092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:31:17.366101 4098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:31:17.366189 4092 server_base.cc:1034] running on GCE node
W20260929 10:31:17.366278 4097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:31:17.366218 4100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:31:17.366833 4092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:31:17.367620 4092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:31:17.368852 4092 hybrid_clock.cc:648] HybridClock initialized: now 1790677877368836 us; error 34 us; skew 500 ppm
Sep 29 10:31:17 dist-test-slave-9cj5 krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677877, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:31:17.372103 4092 init.cc:377] Logged in from keytab as kudu/127.20.92.3@KRBTEST.COM (short username kudu)
I20260929 10:31:17.372998 4092 webserver.cc:460] Webserver started at http://127.20.92.3:44113/ using document root <none> and password file <none>
I20260929 10:31:17.373538 4092 fs_manager.cc:359] Metadata directory not provided
I20260929 10:31:17.373597 4092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:31:17.373842 4092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:31:17.376011 4092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/data/instance:
uuid: "38154f3b4e3f4ab18eeadc2377d821e7"
format_stamp: "Formatted at 2026-09-29 10:31:17 on dist-test-slave-9cj5"
server_key: "62d46ad117332aa631abb0449d1cc034"
server_key_iv: "d07e6904a6908e94475faea8c1a34f5b"
server_key_version: "encryptionkey@0"
I20260929 10:31:17.376545 4092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance:
uuid: "38154f3b4e3f4ab18eeadc2377d821e7"
format_stamp: "Formatted at 2026-09-29 10:31:17 on dist-test-slave-9cj5"
server_key: "62d46ad117332aa631abb0449d1cc034"
server_key_iv: "d07e6904a6908e94475faea8c1a34f5b"
server_key_version: "encryptionkey@0"
I20260929 10:31:17.380719 4092 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 10:31:17.384002 4107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:31:17.385552 4092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 10:31:17.385682 4092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "38154f3b4e3f4ab18eeadc2377d821e7"
format_stamp: "Formatted at 2026-09-29 10:31:17 on dist-test-slave-9cj5"
server_key: "62d46ad117332aa631abb0449d1cc034"
server_key_iv: "d07e6904a6908e94475faea8c1a34f5b"
server_key_version: "encryptionkey@0"
I20260929 10:31:17.385906 4092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:31:17.410967 4092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:31:17.415510 4092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:31:17.415825 4092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:31:17.416494 4092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:31:17.417438 4092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:31:17.417486 4092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:31:17.417557 4092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:31:17.417595 4092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:31:17.427167 4092 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:35189
I20260929 10:31:17.427292 4220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:35189 every 8 connection(s)
I20260929 10:31:17.428181 4092 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:31:17.432848 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 4092
I20260929 10:31:17.433003 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance
I20260929 10:31:17.433291 20848 external_mini_cluster.cc:1428] Setting key 48fe40fb3d19008c1b819a6eb736ea1e
Sep 29 10:31:17 dist-test-slave-9cj5 krb5kdc[3717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677877, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:31:17.443826 3801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:17.430188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:55455 (local address 127.20.92.62:41227)
0929 10:31:17.430331 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 10:31:17.430335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:31:17.431252 (+ 917us) server_negotiation.cc:407] Connection header received
0929 10:31:17.432705 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:31:17.432708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:31:17.432770 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:31:17.432873 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:31:17.434366 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:17.435232 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:17.436582 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:17.436809 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:17.436927 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:17.439878 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:31:17.439881 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:31:17.439884 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:31:17.439913 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:31:17.441427 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:31:17.442150 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:31:17.442154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:31:17.442156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:31:17.442296 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:31:17.442680 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:31:17.442683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:31:17.442685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:31:17.442883 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:31:17.442997 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 10:31:17.443524 (+ 527us) server_negotiation.cc:299] Negotiation successful
0929 10:31:17.443656 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 10:31:17.444674 4223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:17.430579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.3:55455)
0929 10:31:17.431060 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:17.431130 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 10:31:17.432474 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:31:17.433042 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:31:17.433049 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:31:17.433501 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:31:17.434167 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:17.434189 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:17.435364 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:17.435366 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:17.436454 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:17.436464 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:17.436966 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:17.436971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:17.437189 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:17.437954 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:31:17.437973 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:31:17.439660 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:31:17.441577 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:31:17.441584 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:31:17.441602 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:31:17.442034 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:31:17.442413 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:31:17.442416 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:31:17.442418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:31:17.442571 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:31:17.442995 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:31:17.443002 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:31:17.443276 (+ 274us) client_negotiation.cc:769] Sending connection context
0929 10:31:17.443469 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 10:31:17.443829 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":119,"threads_started":1}
I20260929 10:31:17.446085 4221 heartbeater.cc:347] Connected to a master server at 127.20.92.62:41227
I20260929 10:31:17.446400 4221 heartbeater.cc:464] Registering TS with master...
I20260929 10:31:17.447135 4221 heartbeater.cc:511] Master 127.20.92.62:41227 requested a full tablet report, sending...
I20260929 10:31:17.448364 3762 ts_manager.cc:194] Registered new tserver with Master: 38154f3b4e3f4ab18eeadc2377d821e7 (127.20.92.3:35189)
I20260929 10:31:17.448964 3762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.3@KRBTEST.COM'} at 127.20.92.3:55455
I20260929 10:31:17.460534 20848 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:31:17.470638 3801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:17.462766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59510 (local address 127.20.92.62:41227)
0929 10:31:17.462922 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 10:31:17.462936 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 10:31:17.463137 (+ 201us) server_negotiation.cc:407] Connection header received
0929 10:31:17.463234 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:31:17.463237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:31:17.463287 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:31:17.463349 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:31:17.464084 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:17.464618 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:17.465459 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:17.465702 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:17.465866 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:17.467331 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:31:17.467334 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:31:17.467336 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:31:17.467361 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:31:17.468841 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:31:17.469243 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:31:17.469246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:31:17.469247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:31:17.469318 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:31:17.469654 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:31:17.469656 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:31:17.469657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:31:17.469970 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:31:17.470078 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 10:31:17.470338 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 10:31:17.470433 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 10:31:17.473958 3762 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:31:17.477062 3762 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:31:17.494154 4233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:17.486933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:44711 (local address 127.0.0.1:47264)
0929 10:31:17.488363 (+ 1430us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:17.488551 (+ 188us) client_negotiation.cc:174] Beginning negotiation
0929 10:31:17.488629 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:31:17.489357 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:31:17.489360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:31:17.489379 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:31:17.489470 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:17.489477 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:17.490965 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:17.490968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:17.492314 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:17.492324 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:17.493420 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:17.493423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:17.493524 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:17.493571 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 10:31:17.493807 (+ 236us) client_negotiation.cc:240] Negotiation successful
0929 10:31:17.493957 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1365,"mutex_wait_us":84,"thread_start_us":138,"threads_started":1}
I20260929 10:31:17.494499 4234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:17.487358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37626 (local address 127.20.92.3:35189)
0929 10:31:17.488225 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0929 10:31:17.488240 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:31:17.488267 (+ 27us) server_negotiation.cc:407] Connection header received
0929 10:31:17.488355 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:31:17.488364 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:31:17.488625 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:31:17.488794 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:31:17.489117 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:17.490345 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:17.492729 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:17.493524 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:17.493639 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:17.493735 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 10:31:17.494235 (+ 500us) server_negotiation.cc:299] Negotiation successful
0929 10:31:17.494360 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743,"thread_start_us":124,"threads_started":1}
I20260929 10:31:17.494154 4235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:17.486922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:35189 (local address 127.0.0.1:37626)
0929 10:31:17.487907 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:17.487970 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 10:31:17.488108 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:31:17.488791 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:31:17.488799 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:31:17.488823 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:31:17.488973 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:17.488980 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:17.490471 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:17.490475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:17.492574 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:17.492590 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:17.493722 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:17.493728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:17.493848 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:17.493871 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 10:31:17.493948 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 10:31:17.494007 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"mutex_wait_us":61,"thread_start_us":84,"threads_started":1}
I20260929 10:31:17.494866 4236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:17.487794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47264 (local address 127.20.92.2:44711)
0929 10:31:17.488775 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0929 10:31:17.488788 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 10:31:17.488809 (+ 21us) server_negotiation.cc:407] Connection header received
0929 10:31:17.488877 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:31:17.488881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:31:17.489064 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:31:17.489209 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:31:17.489620 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:17.490825 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:17.492441 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:17.493267 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:17.493389 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:17.493551 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 10:31:17.493845 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 10:31:17.493950 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":863,"thread_start_us":124,"threads_started":1}
I20260929 10:31:17.495477 4232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:17.487438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:44141 (local address 127.0.0.1:49548)
0929 10:31:17.488394 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:17.488407 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:31:17.488486 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:31:17.489047 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:31:17.489051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:31:17.489070 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:31:17.489184 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:17.489192 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:17.490852 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:17.490943 (+ 91us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:17.492578 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:17.492589 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:17.495039 (+ 2450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:17.495044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:17.495148 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:17.495182 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 10:31:17.495262 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 10:31:17.495334 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"mutex_wait_us":50,"thread_start_us":100,"threads_started":1}
I20260929 10:31:17.495786 4237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:17.488007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49548 (local address 127.20.92.1:44141)
0929 10:31:17.488578 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0929 10:31:17.488585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:31:17.488608 (+ 23us) server_negotiation.cc:407] Connection header received
0929 10:31:17.488687 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:31:17.488695 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:31:17.488885 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:31:17.489047 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:31:17.489323 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:17.490706 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:17.493846 (+ 3140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:17.494712 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:17.494818 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:17.494970 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 10:31:17.495303 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 10:31:17.495407 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":151,"threads_started":1}
I20260929 10:31:17.497951 4155 tablet_service.cc:1467] Processing CreateTablet for tablet 44101b844cce486eade7331997e43795 (DEFAULT_TABLE table=test-table [id=ccf55e368eaf4dd58f7b1ac6f58b3a27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:31:17.498250 4021 tablet_service.cc:1467] Processing CreateTablet for tablet 44101b844cce486eade7331997e43795 (DEFAULT_TABLE table=test-table [id=ccf55e368eaf4dd58f7b1ac6f58b3a27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:31:17.498286 3887 tablet_service.cc:1467] Processing CreateTablet for tablet 44101b844cce486eade7331997e43795 (DEFAULT_TABLE table=test-table [id=ccf55e368eaf4dd58f7b1ac6f58b3a27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:31:17.499409 4155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44101b844cce486eade7331997e43795. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:31:17.499428 3887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44101b844cce486eade7331997e43795. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:31:17.499433 4021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44101b844cce486eade7331997e43795. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:31:17.506771 4238 tablet_bootstrap.cc:492] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7: Bootstrap starting.
I20260929 10:31:17.508551 4240 tablet_bootstrap.cc:492] T 44101b844cce486eade7331997e43795 P f32320fb1e1a4b248864f8b847cadfe9: Bootstrap starting.
I20260929 10:31:17.509836 4238 tablet_bootstrap.cc:654] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7: Neither blocks nor log segments found. Creating new log.
I20260929 10:31:17.511346 4240 tablet_bootstrap.cc:654] T 44101b844cce486eade7331997e43795 P f32320fb1e1a4b248864f8b847cadfe9: Neither blocks nor log segments found. Creating new log.
I20260929 10:31:17.511823 4238 log.cc:824] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7: Log is configured to *not* fsync() on all Append() calls
I20260929 10:31:17.512113 4240 log.cc:824] T 44101b844cce486eade7331997e43795 P f32320fb1e1a4b248864f8b847cadfe9: Log is configured to *not* fsync() on all Append() calls
I20260929 10:31:17.512931 4239 tablet_bootstrap.cc:492] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13: Bootstrap starting.
I20260929 10:31:17.514357 4240 tablet_bootstrap.cc:492] T 44101b844cce486eade7331997e43795 P f32320fb1e1a4b248864f8b847cadfe9: No bootstrap required, opened a new log
I20260929 10:31:17.514358 4238 tablet_bootstrap.cc:492] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7: No bootstrap required, opened a new log
I20260929 10:31:17.514588 4240 ts_tablet_manager.cc:1397] T 44101b844cce486eade7331997e43795 P f32320fb1e1a4b248864f8b847cadfe9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 10:31:17.514599 4238 ts_tablet_manager.cc:1397] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 10:31:17.515221 4239 tablet_bootstrap.cc:654] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13: Neither blocks nor log segments found. Creating new log.
I20260929 10:31:17.517220 4239 log.cc:824] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13: Log is configured to *not* fsync() on all Append() calls
I20260929 10:31:17.519286 4239 tablet_bootstrap.cc:492] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13: No bootstrap required, opened a new log
I20260929 10:31:17.519186 4240 raft_consensus.cc:348] T 44101b844cce486eade7331997e43795 P f32320fb1e1a4b248864f8b847cadfe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 35189 } } peers { permanent_uuid: "f32320fb1e1a4b248864f8b847cadfe9" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44711 } } peers { permanent_uuid: "842f6d386b2849079c7ce98a3e594a13" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44141 } }
I20260929 10:31:17.519186 4238 raft_consensus.cc:348] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 35189 } } peers { permanent_uuid: "f32320fb1e1a4b248864f8b847cadfe9" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44711 } } peers { permanent_uuid: "842f6d386b2849079c7ce98a3e594a13" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44141 } }
I20260929 10:31:17.519538 4240 raft_consensus.cc:374] T 44101b844cce486eade7331997e43795 P f32320fb1e1a4b248864f8b847cadfe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:31:17.519557 4239 ts_tablet_manager.cc:1397] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260929 10:31:17.519552 4238 raft_consensus.cc:374] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:31:17.519753 4238 raft_consensus.cc:729] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38154f3b4e3f4ab18eeadc2377d821e7, State: Initialized, Role: FOLLOWER
I20260929 10:31:17.519748 4240 raft_consensus.cc:729] T 44101b844cce486eade7331997e43795 P f32320fb1e1a4b248864f8b847cadfe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f32320fb1e1a4b248864f8b847cadfe9, State: Initialized, Role: FOLLOWER
I20260929 10:31:17.522346 4240 consensus_queue.cc:260] T 44101b844cce486eade7331997e43795 P f32320fb1e1a4b248864f8b847cadfe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 35189 } } peers { permanent_uuid: "f32320fb1e1a4b248864f8b847cadfe9" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44711 } } peers { permanent_uuid: "842f6d386b2849079c7ce98a3e594a13" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44141 } }
I20260929 10:31:17.522346 4238 consensus_queue.cc:260] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 35189 } } peers { permanent_uuid: "f32320fb1e1a4b248864f8b847cadfe9" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44711 } } peers { permanent_uuid: "842f6d386b2849079c7ce98a3e594a13" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44141 } }
I20260929 10:31:17.523578 4238 ts_tablet_manager.cc:1428] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260929 10:31:17.523578 4240 ts_tablet_manager.cc:1428] T 44101b844cce486eade7331997e43795 P f32320fb1e1a4b248864f8b847cadfe9: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260929 10:31:17.523983 4087 heartbeater.cc:503] Master 127.20.92.62:41227 was elected leader, sending a full tablet report...
I20260929 10:31:17.523991 4221 heartbeater.cc:503] Master 127.20.92.62:41227 was elected leader, sending a full tablet report...
I20260929 10:31:17.527343 4239 raft_consensus.cc:348] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 35189 } } peers { permanent_uuid: "f32320fb1e1a4b248864f8b847cadfe9" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44711 } } peers { permanent_uuid: "842f6d386b2849079c7ce98a3e594a13" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44141 } }
I20260929 10:31:17.528002 4239 raft_consensus.cc:374] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:31:17.528090 4239 raft_consensus.cc:729] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842f6d386b2849079c7ce98a3e594a13, State: Initialized, Role: FOLLOWER
I20260929 10:31:17.528708 4239 consensus_queue.cc:260] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 35189 } } peers { permanent_uuid: "f32320fb1e1a4b248864f8b847cadfe9" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44711 } } peers { permanent_uuid: "842f6d386b2849079c7ce98a3e594a13" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44141 } }
I20260929 10:31:17.529619 3953 heartbeater.cc:503] Master 127.20.92.62:41227 was elected leader, sending a full tablet report...
I20260929 10:31:17.530251 4239 ts_tablet_manager.cc:1428] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13: Time spent starting tablet: real 0.011s user 0.005s sys 0.006s
W20260929 10:31:17.665076 4088 tablet.cc:2369] T 44101b844cce486eade7331997e43795 P f32320fb1e1a4b248864f8b847cadfe9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:31:17.669893 3954 tablet.cc:2369] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:31:17.679263 4222 tablet.cc:2369] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:31:17.692510 4244 raft_consensus.cc:482] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:31:17.692826 4244 raft_consensus.cc:504] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 35189 } } peers { permanent_uuid: "f32320fb1e1a4b248864f8b847cadfe9" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44711 } } peers { permanent_uuid: "842f6d386b2849079c7ce98a3e594a13" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44141 } }
I20260929 10:31:17.694113 4244 leader_election.cc:288] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f32320fb1e1a4b248864f8b847cadfe9 (127.20.92.2:44711), 842f6d386b2849079c7ce98a3e594a13 (127.20.92.1:44141)
I20260929 10:31:17.698670 4223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:17.694443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:44711 (local address 127.20.92.3:37843)
0929 10:31:17.694582 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:17.694600 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 10:31:17.694706 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:31:17.695025 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:31:17.695028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:31:17.695080 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:31:17.695241 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:17.695250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:17.696232 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:17.696235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:17.697184 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:17.697193 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:17.698121 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:17.698125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:17.698260 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:17.698295 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 10:31:17.698391 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 10:31:17.698477 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 10:31:17.698685 4236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:17.694525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:37843 (local address 127.20.92.2:44711)
0929 10:31:17.694714 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 10:31:17.694720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:31:17.694739 (+ 19us) server_negotiation.cc:407] Connection header received
0929 10:31:17.694811 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:31:17.694814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:31:17.694867 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:31:17.694964 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:31:17.695404 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:17.696098 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:17.697331 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:17.697912 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:17.698350 (+ 438us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:17.698416 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 10:31:17.698458 (+ 42us) server_negotiation.cc:299] Negotiation successful
0929 10:31:17.698521 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 10:31:17.699282 4237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:17.694626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:43635 (local address 127.20.92.1:44141)
0929 10:31:17.694754 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 10:31:17.694759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:31:17.695089 (+ 330us) server_negotiation.cc:407] Connection header received
0929 10:31:17.695201 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:31:17.695206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:31:17.695263 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:31:17.695353 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:31:17.695704 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:17.696284 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:17.697462 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:17.698538 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:17.698664 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:17.698754 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 10:31:17.699036 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 10:31:17.699151 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 10:31:17.699559 4041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44101b844cce486eade7331997e43795" candidate_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f32320fb1e1a4b248864f8b847cadfe9" is_pre_election: true
I20260929 10:31:17.699673 4247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:17.694451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:44141 (local address 127.20.92.3:43635)
0929 10:31:17.694960 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:17.694974 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:31:17.695112 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:31:17.695401 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:31:17.695404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:31:17.695422 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:31:17.695534 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:17.695541 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:17.696408 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:17.696411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:17.697310 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:17.697323 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:17.698717 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:17.698722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:17.698845 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:17.698878 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 10:31:17.699454 (+ 576us) client_negotiation.cc:240] Negotiation successful
0929 10:31:17.699556 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":139,"spinlock_wait_cycles":3840,"thread_start_us":127,"threads_started":1}
I20260929 10:31:17.699941 4041 raft_consensus.cc:2454] T 44101b844cce486eade7331997e43795 P f32320fb1e1a4b248864f8b847cadfe9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38154f3b4e3f4ab18eeadc2377d821e7 in term 0.
I20260929 10:31:17.700417 3907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44101b844cce486eade7331997e43795" candidate_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842f6d386b2849079c7ce98a3e594a13" is_pre_election: true
I20260929 10:31:17.700649 4110 leader_election.cc:302] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38154f3b4e3f4ab18eeadc2377d821e7, f32320fb1e1a4b248864f8b847cadfe9; no voters:
I20260929 10:31:17.700718 3907 raft_consensus.cc:2454] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38154f3b4e3f4ab18eeadc2377d821e7 in term 0.
I20260929 10:31:17.700958 4244 raft_consensus.cc:2781] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:31:17.701166 4244 raft_consensus.cc:482] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:31:17.701258 4244 raft_consensus.cc:3037] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:31:17.702414 4244 raft_consensus.cc:504] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 35189 } } peers { permanent_uuid: "f32320fb1e1a4b248864f8b847cadfe9" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44711 } } peers { permanent_uuid: "842f6d386b2849079c7ce98a3e594a13" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44141 } }
I20260929 10:31:17.702824 4244 leader_election.cc:288] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7 [CANDIDATE]: Term 1 election: Requested vote from peers f32320fb1e1a4b248864f8b847cadfe9 (127.20.92.2:44711), 842f6d386b2849079c7ce98a3e594a13 (127.20.92.1:44141)
I20260929 10:31:17.703315 4041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44101b844cce486eade7331997e43795" candidate_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f32320fb1e1a4b248864f8b847cadfe9"
I20260929 10:31:17.703438 3907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44101b844cce486eade7331997e43795" candidate_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842f6d386b2849079c7ce98a3e594a13"
I20260929 10:31:17.703495 4041 raft_consensus.cc:3037] T 44101b844cce486eade7331997e43795 P f32320fb1e1a4b248864f8b847cadfe9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:31:17.703614 3907 raft_consensus.cc:3037] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:31:17.704694 4041 raft_consensus.cc:2454] T 44101b844cce486eade7331997e43795 P f32320fb1e1a4b248864f8b847cadfe9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38154f3b4e3f4ab18eeadc2377d821e7 in term 1.
I20260929 10:31:17.704810 3907 raft_consensus.cc:2454] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38154f3b4e3f4ab18eeadc2377d821e7 in term 1.
I20260929 10:31:17.705206 4110 leader_election.cc:302] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38154f3b4e3f4ab18eeadc2377d821e7, f32320fb1e1a4b248864f8b847cadfe9; no voters:
I20260929 10:31:17.705415 4244 raft_consensus.cc:2781] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:31:17.705925 4244 raft_consensus.cc:686] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7 [term 1 LEADER]: Becoming Leader. State: Replica: 38154f3b4e3f4ab18eeadc2377d821e7, State: Running, Role: LEADER
I20260929 10:31:17.706346 4244 consensus_queue.cc:237] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 35189 } } peers { permanent_uuid: "f32320fb1e1a4b248864f8b847cadfe9" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44711 } } peers { permanent_uuid: "842f6d386b2849079c7ce98a3e594a13" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44141 } }
I20260929 10:31:17.710048 3762 catalog_manager.cc:5495] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38154f3b4e3f4ab18eeadc2377d821e7 (127.20.92.3). New cstate: current_term: 1 leader_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 35189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f32320fb1e1a4b248864f8b847cadfe9" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "842f6d386b2849079c7ce98a3e594a13" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44141 } health_report { overall_health: UNKNOWN } } }
I20260929 10:31:17.742805 4234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:17.738395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37642 (local address 127.20.92.3:35189)
0929 10:31:17.738558 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 10:31:17.738563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:31:17.738582 (+ 19us) server_negotiation.cc:407] Connection header received
0929 10:31:17.738740 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:31:17.738742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:31:17.738805 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:31:17.738929 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:31:17.739225 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:17.739782 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:17.740759 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:17.741002 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:17.741129 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:17.741505 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:31:17.742254 (+ 749us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:31:17.742428 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 10:31:17.742504 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 10:31:17.742575 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 10:31:17.750506 4041 raft_consensus.cc:1261] T 44101b844cce486eade7331997e43795 P f32320fb1e1a4b248864f8b847cadfe9 [term 1 FOLLOWER]: Refusing update from remote peer 38154f3b4e3f4ab18eeadc2377d821e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:31:17.750546 3907 raft_consensus.cc:1261] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13 [term 1 FOLLOWER]: Refusing update from remote peer 38154f3b4e3f4ab18eeadc2377d821e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:31:17.751350 4248 consensus_queue.cc:1035] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f32320fb1e1a4b248864f8b847cadfe9" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:31:17.751590 4244 consensus_queue.cc:1035] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "842f6d386b2849079c7ce98a3e594a13" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:31:17.759404 4251 mvcc.cc:205] Tried to move back new op lower bound from 7334616587257769984 to 7334616587087941632. Current Snapshot: MvccSnapshot[applied={T|T < 7334616587257769984}]
I20260929 10:31:17.763509 4252 mvcc.cc:205] Tried to move back new op lower bound from 7334616587257769984 to 7334616587087941632. Current Snapshot: MvccSnapshot[applied={T|T < 7334616587257769984}]
I20260929 10:31:17.763940 4253 mvcc.cc:205] Tried to move back new op lower bound from 7334616587257769984 to 7334616587087941632. Current Snapshot: MvccSnapshot[applied={T|T < 7334616587257769984 or (T in {7334616587257769984})}]
I20260929 10:31:17.768455 3762 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:31:17.769709 3762 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:31:17.773116 3762 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005: Sending DeleteTablet for 3 replicas of tablet 44101b844cce486eade7331997e43795
I20260929 10:31:17.774245 4155 tablet_service.cc:1514] Processing DeleteTablet for tablet 44101b844cce486eade7331997e43795 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:31:17 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:37626
I20260929 10:31:17.774243 4021 tablet_service.cc:1514] Processing DeleteTablet for tablet 44101b844cce486eade7331997e43795 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:31:17 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:47264
I20260929 10:31:17.774283 3887 tablet_service.cc:1514] Processing DeleteTablet for tablet 44101b844cce486eade7331997e43795 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:31:17 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:49548
I20260929 10:31:17.774874 4262 tablet_replica.cc:330] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7: stopping tablet replica
I20260929 10:31:17.775004 4263 tablet_replica.cc:330] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13: stopping tablet replica
I20260929 10:31:17.775375 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 3824
I20260929 10:31:17.775476 4262 raft_consensus.cc:2229] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:31:17.775476 4263 raft_consensus.cc:2229] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:31:17.775934 4262 raft_consensus.cc:2258] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:31:17.775962 4263 raft_consensus.cc:2258] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:31:17.776521 4261 tablet_replica.cc:330] T 44101b844cce486eade7331997e43795 P f32320fb1e1a4b248864f8b847cadfe9: stopping tablet replica
I20260929 10:31:17.776773 4261 raft_consensus.cc:2229] T 44101b844cce486eade7331997e43795 P f32320fb1e1a4b248864f8b847cadfe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:31:17.777015 4261 raft_consensus.cc:2258] T 44101b844cce486eade7331997e43795 P f32320fb1e1a4b248864f8b847cadfe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:31:17.778788 4261 ts_tablet_manager.cc:1905] T 44101b844cce486eade7331997e43795 P f32320fb1e1a4b248864f8b847cadfe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:31:17.778788 4262 ts_tablet_manager.cc:1905] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:31:17.782398 4261 ts_tablet_manager.cc:1918] T 44101b844cce486eade7331997e43795 P f32320fb1e1a4b248864f8b847cadfe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:31:17.782413 4262 ts_tablet_manager.cc:1918] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:31:17.782626 4262 log.cc:1196] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/wal/wals/44101b844cce486eade7331997e43795
I20260929 10:31:17.782847 4261 log.cc:1196] T 44101b844cce486eade7331997e43795 P f32320fb1e1a4b248864f8b847cadfe9: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/wal/wals/44101b844cce486eade7331997e43795
I20260929 10:31:17.783142 4262 ts_tablet_manager.cc:1939] T 44101b844cce486eade7331997e43795 P 38154f3b4e3f4ab18eeadc2377d821e7: Deleting consensus metadata
I20260929 10:31:17.783272 4261 ts_tablet_manager.cc:1939] T 44101b844cce486eade7331997e43795 P f32320fb1e1a4b248864f8b847cadfe9: Deleting consensus metadata
I20260929 10:31:17.785727 3749 catalog_manager.cc:4841] TS 38154f3b4e3f4ab18eeadc2377d821e7 (127.20.92.3:35189): tablet 44101b844cce486eade7331997e43795 (table test-table [id=ccf55e368eaf4dd58f7b1ac6f58b3a27]) successfully deleted
I20260929 10:31:17.785729 3748 catalog_manager.cc:4841] TS f32320fb1e1a4b248864f8b847cadfe9 (127.20.92.2:44711): tablet 44101b844cce486eade7331997e43795 (table test-table [id=ccf55e368eaf4dd58f7b1ac6f58b3a27]) successfully deleted
W20260929 10:31:17.787439 3746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.92.1:44141 (error 108)
I20260929 10:31:17.788481 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 3958
I20260929 10:31:17.788781 4232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:17.788447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:44141 (local address 127.0.0.1:49564)
0929 10:31:17.788614 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:17.788689 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.1:44141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 10:31:17.789682 3746 catalog_manager.cc:4584] TS 842f6d386b2849079c7ce98a3e594a13 (127.20.92.1:44141): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 44101b844cce486eade7331997e43795: Network error: Client connection negotiation failed: client connection to 127.20.92.1:44141: connect: Connection refused (error 111)
I20260929 10:31:17.797123 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 4092
I20260929 10:31:17.809237 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 3730
I20260929 10:31:37.821583 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:41227
--webserver_interface=127.20.92.62
--webserver_port=46847
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:41227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:31:37.935714 4265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:31:37.936038 4265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:31:37.936084 4265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:31:37.940191 4265 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 10:31:37.940249 4265 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 10:31:37.940267 4265 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 10:31:37.940281 4265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:31:37.940299 4265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:31:37.940375 4265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:31:37.940402 4265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:31:37.940434 4265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:31:37.945928 4265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:41227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:41227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=46847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:31:37.947080 4265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:31:37.948164 4265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:31:37.963102 4274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:31:37.963161 4272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:31:37.963102 4271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:31:37.963533 4265 server_base.cc:1034] running on GCE node
I20260929 10:31:37.964159 4265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:31:37.965437 4265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:31:37.966622 4265 hybrid_clock.cc:648] HybridClock initialized: now 1790677897966615 us; error 74 us; skew 500 ppm
Sep 29 10:31:37 dist-test-slave-9cj5 krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677897, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:31:37.974241 4265 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:31:37.975279 4265 webserver.cc:460] Webserver started at http://127.20.92.62:46847/ using document root <none> and password file <none>
I20260929 10:31:37.975919 4265 fs_manager.cc:359] Metadata directory not provided
I20260929 10:31:37.975997 4265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:31:37.980224 4265 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:31:37.982590 4281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:31:37.983975 4265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:31:37.984156 4265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "f0706ae6bb3b47318e36811afe109005"
format_stamp: "Formatted at 2026-09-29 10:31:16 on dist-test-slave-9cj5"
server_key: "c3b77caac26bdc748e973636cd83034d"
server_key_iv: "eab84ab7f44e19c62da80c319b5dbc23"
server_key_version: "encryptionkey@0"
I20260929 10:31:37.984807 4265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:31:38.029531 4265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:31:38.032867 4265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:31:38.033100 4265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:31:38.041006 4265 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:41227
I20260929 10:31:38.041302 4333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:41227 every 8 connection(s)
I20260929 10:31:38.042236 4265 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:31:38.046991 4334 sys_catalog.cc:263] Verifying existing consensus state
I20260929 10:31:38.048823 4334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005: Bootstrap starting.
I20260929 10:31:38.049319 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 4265
I20260929 10:31:38.049839 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:44141
--local_ip_for_outbound_sockets=127.20.92.1
--tserver_master_addrs=127.20.92.62:41227
--webserver_port=41693
--webserver_interface=127.20.92.1
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:31:38.066370 4334 log.cc:824] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005: Log is configured to *not* fsync() on all Append() calls
I20260929 10:31:38.075285 4334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:31:38.075762 4334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005: Bootstrap complete.
I20260929 10:31:38.079959 4334 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0706ae6bb3b47318e36811afe109005" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 41227 } }
I20260929 10:31:38.080351 4334 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0706ae6bb3b47318e36811afe109005, State: Initialized, Role: FOLLOWER
I20260929 10:31:38.080849 4334 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0706ae6bb3b47318e36811afe109005" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 41227 } }
I20260929 10:31:38.080966 4334 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:31:38.081065 4334 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:31:38.081192 4334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [term 1 FOLLOWER]: Advancing to term 2
I20260929 10:31:38.083216 4334 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0706ae6bb3b47318e36811afe109005" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 41227 } }
I20260929 10:31:38.083565 4334 leader_election.cc:302] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0706ae6bb3b47318e36811afe109005; no voters:
I20260929 10:31:38.083928 4334 leader_election.cc:288] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 10:31:38.084018 4338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 10:31:38.084234 4338 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [term 2 LEADER]: Becoming Leader. State: Replica: f0706ae6bb3b47318e36811afe109005, State: Running, Role: LEADER
I20260929 10:31:38.084584 4338 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0706ae6bb3b47318e36811afe109005" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 41227 } }
I20260929 10:31:38.085196 4334 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:31:38.085582 4338 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0706ae6bb3b47318e36811afe109005. Latest consensus state: current_term: 2 leader_uuid: "f0706ae6bb3b47318e36811afe109005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0706ae6bb3b47318e36811afe109005" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 41227 } } }
I20260929 10:31:38.085726 4338 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [sys.catalog]: This master's current role is: LEADER
I20260929 10:31:38.086099 4339 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f0706ae6bb3b47318e36811afe109005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0706ae6bb3b47318e36811afe109005" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 41227 } } }
I20260929 10:31:38.086282 4339 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [sys.catalog]: This master's current role is: LEADER
I20260929 10:31:38.086577 4342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:31:38.091909 4342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:31:38.092757 4342 catalog_manager.cc:1259] Loaded cluster ID: 10bcea7b0f30465b94c67a065b23a4c4
I20260929 10:31:38.092818 4342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:31:38.095361 4342 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:31:38.096292 4342 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005: Loaded TSK: 0
I20260929 10:31:38.109891 4342 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005: Generated new TSK 1
I20260929 10:31:38.110165 4342 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 10:31:38.174738 4336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:31:38.175082 4336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:31:38.175208 4336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:31:38.179988 4336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:31:38.180065 4336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:31:38.180130 4336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:31:38.185802 4336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:44141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=41693
--tserver_master_addrs=127.20.92.62:41227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:31:38.186971 4336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:31:38.188144 4336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:31:38.199237 4362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:31:38.199301 4361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:31:38.199323 4364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:31:38.199873 4336 server_base.cc:1034] running on GCE node
I20260929 10:31:38.200430 4336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:31:38.201313 4336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:31:38.202529 4336 hybrid_clock.cc:648] HybridClock initialized: now 1790677898202492 us; error 45 us; skew 500 ppm
Sep 29 10:31:38 dist-test-slave-9cj5 krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677898, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:31:38.206136 4336 init.cc:377] Logged in from keytab as kudu/127.20.92.1@KRBTEST.COM (short username kudu)
I20260929 10:31:38.207284 4336 webserver.cc:460] Webserver started at http://127.20.92.1:41693/ using document root <none> and password file <none>
I20260929 10:31:38.208029 4336 fs_manager.cc:359] Metadata directory not provided
I20260929 10:31:38.208145 4336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:31:38.213492 4336 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:31:38.216394 4371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:31:38.217978 4336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:31:38.218106 4336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "842f6d386b2849079c7ce98a3e594a13"
format_stamp: "Formatted at 2026-09-29 10:31:16 on dist-test-slave-9cj5"
server_key: "e5da3ca21baae1b61c8b7a19a93aaf4a"
server_key_iv: "a676b4c99fd4bb9fff7214bdc75124ce"
server_key_version: "encryptionkey@0"
I20260929 10:31:38.218807 4336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:31:38.237337 4336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:31:38.241166 4336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:31:38.241361 4336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:31:38.242074 4336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:31:38.243392 4378 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 10:31:38.247632 4336 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 10:31:38.247710 4336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260929 10:31:38.247784 4336 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 10:31:38.249411 4336 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 10:31:38.249456 4336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 10:31:38.249790 4378 tablet_bootstrap.cc:492] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13: Bootstrap starting.
I20260929 10:31:38.259760 4336 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:44141
I20260929 10:31:38.259864 4485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:44141 every 8 connection(s)
I20260929 10:31:38.260828 4336 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
I20260929 10:31:38.266978 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 4336
I20260929 10:31:38.267506 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:44711
--local_ip_for_outbound_sockets=127.20.92.2
--tserver_master_addrs=127.20.92.62:41227
--webserver_port=45625
--webserver_interface=127.20.92.2
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:31:38.268962 4378 log.cc:824] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13: Log is configured to *not* fsync() on all Append() calls
Sep 29 10:31:38 dist-test-slave-9cj5 krb5kdc[3717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677898, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:31:38.277639 4489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:38.263099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.1:45243)
0929 10:31:38.263758 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:38.263800 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 10:31:38.264619 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:31:38.265711 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:31:38.265719 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:31:38.266170 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:31:38.266618 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:38.266632 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.268104 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.268109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:38.269114 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:38.269129 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.270085 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.270088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:38.270274 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:38.270894 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:31:38.270910 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:31:38.272419 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:31:38.274580 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:31:38.274585 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:31:38.274598 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:31:38.274857 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:31:38.275215 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:31:38.275218 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:31:38.275220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:31:38.275352 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:31:38.275932 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:31:38.275938 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:31:38.276200 (+ 262us) client_negotiation.cc:769] Sending connection context
0929 10:31:38.276391 (+ 191us) client_negotiation.cc:240] Negotiation successful
0929 10:31:38.276731 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":149,"threads_started":1}
I20260929 10:31:38.277622 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:38.263054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:45243 (local address 127.20.92.62:41227)
0929 10:31:38.263442 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0929 10:31:38.263457 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:31:38.263926 (+ 469us) server_negotiation.cc:407] Connection header received
0929 10:31:38.265029 (+ 1103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:31:38.265040 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:31:38.265334 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:31:38.265659 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:31:38.266800 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.267969 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.269276 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.269936 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.270123 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:38.272621 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:31:38.272626 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:31:38.272638 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:31:38.272665 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:31:38.274438 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:31:38.274976 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:31:38.274982 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:31:38.274989 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:31:38.275110 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:31:38.275455 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:31:38.275458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:31:38.275460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:31:38.275809 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:31:38.275960 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 10:31:38.276502 (+ 542us) server_negotiation.cc:299] Negotiation successful
0929 10:31:38.276772 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":135,"threads_started":1}
I20260929 10:31:38.279038 4378 tablet_bootstrap.cc:492] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:31:38.279543 4378 tablet_bootstrap.cc:492] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13: Bootstrap complete.
I20260929 10:31:38.280570 4486 heartbeater.cc:347] Connected to a master server at 127.20.92.62:41227
I20260929 10:31:38.280810 4486 heartbeater.cc:464] Registering TS with master...
I20260929 10:31:38.281000 4378 ts_tablet_manager.cc:1397] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13: Time spent bootstrapping tablet: real 0.031s user 0.028s sys 0.000s
I20260929 10:31:38.281366 4486 heartbeater.cc:511] Master 127.20.92.62:41227 requested a full tablet report, sending...
I20260929 10:31:38.283797 4298 ts_manager.cc:194] Registered new tserver with Master: 842f6d386b2849079c7ce98a3e594a13 (127.20.92.1:44141)
I20260929 10:31:38.284067 4378 raft_consensus.cc:348] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 35189 } } peers { permanent_uuid: "f32320fb1e1a4b248864f8b847cadfe9" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44711 } } peers { permanent_uuid: "842f6d386b2849079c7ce98a3e594a13" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44141 } }
I20260929 10:31:38.284649 4378 raft_consensus.cc:729] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842f6d386b2849079c7ce98a3e594a13, State: Initialized, Role: FOLLOWER
I20260929 10:31:38.285382 4378 consensus_queue.cc:260] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 35189 } } peers { permanent_uuid: "f32320fb1e1a4b248864f8b847cadfe9" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44711 } } peers { permanent_uuid: "842f6d386b2849079c7ce98a3e594a13" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44141 } }
I20260929 10:31:38.286751 4378 ts_tablet_manager.cc:1428] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260929 10:31:38.288589 4298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.1@KRBTEST.COM'} at 127.20.92.1:45243
I20260929 10:31:38.291330 4486 heartbeater.cc:503] Master 127.20.92.62:41227 was elected leader, sending a full tablet report...
I20260929 10:31:38.295390 4494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:38.288080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41936 (local address 127.20.92.1:44141)
0929 10:31:38.288500 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0929 10:31:38.288512 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 10:31:38.289717 (+ 1205us) server_negotiation.cc:407] Connection header received
0929 10:31:38.289919 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:31:38.289925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:31:38.290490 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:31:38.290648 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:31:38.290946 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.292254 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.293830 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.294568 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.294680 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:38.294813 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 10:31:38.295205 (+ 392us) server_negotiation.cc:299] Negotiation successful
0929 10:31:38.295283 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":106,"threads_started":1}
I20260929 10:31:38.295421 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:38.287891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:44141 (local address 127.0.0.1:41936)
0929 10:31:38.289532 (+ 1641us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:38.289568 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 10:31:38.289728 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:31:38.290673 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:31:38.290679 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:31:38.290704 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:31:38.290825 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:38.290833 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.292375 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.292379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:38.293696 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:38.293707 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.294879 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.294883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:38.295005 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:38.295044 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 10:31:38.295160 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 10:31:38.295282 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1468,"thread_start_us":119,"threads_started":1}
I20260929 10:31:38.296475 4420 tablet_service.cc:1514] Processing DeleteTablet for tablet 44101b844cce486eade7331997e43795 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:31:17 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:41936
I20260929 10:31:38.297469 4495 tablet_replica.cc:330] stopping tablet replica
I20260929 10:31:38.297835 4495 raft_consensus.cc:2229] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:31:38.298089 4495 raft_consensus.cc:2258] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:31:38.299044 4495 ts_tablet_manager.cc:1905] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:31:38.302693 4495 ts_tablet_manager.cc:1918] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:31:38.302773 4495 log.cc:1196] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/wal/wals/44101b844cce486eade7331997e43795
I20260929 10:31:38.303079 4495 ts_tablet_manager.cc:1939] T 44101b844cce486eade7331997e43795 P 842f6d386b2849079c7ce98a3e594a13: Deleting consensus metadata
I20260929 10:31:38.304771 4282 catalog_manager.cc:4841] TS 842f6d386b2849079c7ce98a3e594a13 (127.20.92.1:44141): tablet 44101b844cce486eade7331997e43795 (table test-table [id=ccf55e368eaf4dd58f7b1ac6f58b3a27]) successfully deleted
W20260929 10:31:38.390717 4490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:31:38.391064 4490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:31:38.391148 4490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:31:38.396179 4490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:31:38.396294 4490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:31:38.396399 4490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:31:38.403446 4490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:44711
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=45625
--tserver_master_addrs=127.20.92.62:41227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:31:38.405149 4490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:31:38.406651 4490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:31:38.418009 4501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:31:38.418007 4503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:31:38.418011 4500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:31:38.418324 4490 server_base.cc:1034] running on GCE node
I20260929 10:31:38.418856 4490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:31:38.419611 4490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:31:38.420864 4490 hybrid_clock.cc:648] HybridClock initialized: now 1790677898420843 us; error 40 us; skew 500 ppm
Sep 29 10:31:38 dist-test-slave-9cj5 krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677898, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:31:38.423976 4490 init.cc:377] Logged in from keytab as kudu/127.20.92.2@KRBTEST.COM (short username kudu)
I20260929 10:31:38.425041 4490 webserver.cc:460] Webserver started at http://127.20.92.2:45625/ using document root <none> and password file <none>
I20260929 10:31:38.425592 4490 fs_manager.cc:359] Metadata directory not provided
I20260929 10:31:38.425668 4490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:31:38.430121 4490 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 10:31:38.432412 4510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:31:38.434094 4490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:31:38.434226 4490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "f32320fb1e1a4b248864f8b847cadfe9"
format_stamp: "Formatted at 2026-09-29 10:31:17 on dist-test-slave-9cj5"
server_key: "6ba5ddf9ba44f01fd2ef8fc0f6383f79"
server_key_iv: "3901f4ac528d7646f363b96fb09cb649"
server_key_version: "encryptionkey@0"
I20260929 10:31:38.434897 4490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:31:38.449122 4490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:31:38.452653 4490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:31:38.452821 4490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:31:38.453703 4490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:31:38.454694 4490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:31:38.454744 4490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:31:38.454788 4490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:31:38.454849 4490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:31:38.464570 4490 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:44711
I20260929 10:31:38.464596 4623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:44711 every 8 connection(s)
I20260929 10:31:38.465654 4490 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
I20260929 10:31:38.475739 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 4490
Sep 29 10:31:38 dist-test-slave-9cj5 krb5kdc[3717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677898, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:31:38.476204 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:35189
--local_ip_for_outbound_sockets=127.20.92.3
--tserver_master_addrs=127.20.92.62:41227
--webserver_port=44113
--webserver_interface=127.20.92.3
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:31:38.480433 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:38.467721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:57279 (local address 127.20.92.62:41227)
0929 10:31:38.467914 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 10:31:38.467918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:31:38.468740 (+ 822us) server_negotiation.cc:407] Connection header received
0929 10:31:38.470189 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:31:38.470192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:31:38.470252 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:31:38.470361 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:31:38.471560 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.472103 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.473273 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.473529 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.473667 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:38.476655 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:31:38.476659 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:31:38.476661 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:31:38.476688 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:31:38.478303 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:31:38.478847 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:31:38.478851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:31:38.478853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:31:38.478933 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:31:38.479312 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:31:38.479315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:31:38.479317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:31:38.479506 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:31:38.479608 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 10:31:38.480129 (+ 521us) server_negotiation.cc:299] Negotiation successful
0929 10:31:38.480269 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 10:31:38.481269 4626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:38.468097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.2:57279)
0929 10:31:38.468572 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:38.468614 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 10:31:38.469961 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:31:38.470545 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:31:38.470552 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:31:38.470918 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:31:38.471401 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:38.471414 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.472232 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.472238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:38.473140 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:38.473150 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.473710 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.473714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:38.473914 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:38.474479 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:31:38.474549 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:31:38.476365 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:31:38.478442 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:31:38.478449 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:31:38.478465 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:31:38.478738 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:31:38.479053 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:31:38.479058 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:31:38.479060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:31:38.479208 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:31:38.479618 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:31:38.479627 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:31:38.479885 (+ 258us) client_negotiation.cc:769] Sending connection context
0929 10:31:38.480079 (+ 194us) client_negotiation.cc:240] Negotiation successful
0929 10:31:38.480423 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":108,"threads_started":1}
I20260929 10:31:38.482646 4624 heartbeater.cc:347] Connected to a master server at 127.20.92.62:41227
I20260929 10:31:38.482935 4624 heartbeater.cc:464] Registering TS with master...
I20260929 10:31:38.483582 4624 heartbeater.cc:511] Master 127.20.92.62:41227 requested a full tablet report, sending...
I20260929 10:31:38.484803 4298 ts_manager.cc:194] Registered new tserver with Master: f32320fb1e1a4b248864f8b847cadfe9 (127.20.92.2:44711)
I20260929 10:31:38.485479 4298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.2@KRBTEST.COM'} at 127.20.92.2:57279
W20260929 10:31:38.593801 4627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:31:38.594151 4627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:31:38.594244 4627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:31:38.598798 4627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:31:38.598937 4627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:31:38.599044 4627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:31:38.605055 4627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:35189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=44113
--tserver_master_addrs=127.20.92.62:41227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:31:38.606375 4627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:31:38.607550 4627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:31:38.618223 4633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:31:38.618238 4632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:31:38.618364 4635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:31:38.618773 4627 server_base.cc:1034] running on GCE node
I20260929 10:31:38.619239 4627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:31:38.619956 4627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:31:38.621158 4627 hybrid_clock.cc:648] HybridClock initialized: now 1790677898621140 us; error 34 us; skew 500 ppm
Sep 29 10:31:38 dist-test-slave-9cj5 krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677898, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:31:38.624763 4627 init.cc:377] Logged in from keytab as kudu/127.20.92.3@KRBTEST.COM (short username kudu)
I20260929 10:31:38.625905 4627 webserver.cc:460] Webserver started at http://127.20.92.3:44113/ using document root <none> and password file <none>
I20260929 10:31:38.626638 4627 fs_manager.cc:359] Metadata directory not provided
I20260929 10:31:38.626730 4627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:31:38.630930 4627 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 10:31:38.633266 4642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:31:38.634826 4627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:31:38.634986 4627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "38154f3b4e3f4ab18eeadc2377d821e7"
format_stamp: "Formatted at 2026-09-29 10:31:17 on dist-test-slave-9cj5"
server_key: "62d46ad117332aa631abb0449d1cc034"
server_key_iv: "d07e6904a6908e94475faea8c1a34f5b"
server_key_version: "encryptionkey@0"
I20260929 10:31:38.635571 4627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:31:38.660910 4627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:31:38.664748 4627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:31:38.664990 4627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:31:38.665663 4627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:31:38.666568 4627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:31:38.666640 4627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:31:38.666718 4627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:31:38.666762 4627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:31:38.676054 4627 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:35189
I20260929 10:31:38.676074 4757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:35189 every 8 connection(s)
I20260929 10:31:38.677083 4627 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:31:38.683638 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 4627
Sep 29 10:31:38 dist-test-slave-9cj5 krb5kdc[3717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677898, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:31:38.692602 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:38.678977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:39113 (local address 127.20.92.62:41227)
0929 10:31:38.679121 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 10:31:38.679125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:31:38.680052 (+ 927us) server_negotiation.cc:407] Connection header received
0929 10:31:38.681456 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:31:38.681460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:31:38.681521 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:31:38.681638 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:31:38.682902 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.683423 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.684943 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.685177 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.685295 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:38.688583 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:31:38.688586 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:31:38.688588 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:31:38.688615 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:31:38.690396 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:31:38.690915 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:31:38.690920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:31:38.690922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:31:38.691007 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:31:38.691373 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:31:38.691378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:31:38.691380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:31:38.691561 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:31:38.691670 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 10:31:38.692251 (+ 581us) server_negotiation.cc:299] Negotiation successful
0929 10:31:38.692391 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 10:31:38.693775 4760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:38.679386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.3:39113)
0929 10:31:38.679872 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:38.679917 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 10:31:38.681226 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:31:38.681865 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:31:38.681872 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:31:38.682263 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:31:38.682746 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:38.682759 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.683556 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.683560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:38.684810 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:38.684824 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.685337 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.685341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:38.685520 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:38.686338 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:31:38.686360 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:31:38.688362 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:31:38.690511 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:31:38.690518 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:31:38.690536 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:31:38.690807 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:31:38.691107 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:31:38.691110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:31:38.691112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:31:38.691275 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:31:38.691673 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:31:38.691684 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:31:38.691965 (+ 281us) client_negotiation.cc:769] Sending connection context
0929 10:31:38.692207 (+ 242us) client_negotiation.cc:240] Negotiation successful
0929 10:31:38.692569 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":112,"threads_started":1}
I20260929 10:31:38.693833 4762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:38.684374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39816 (local address 127.20.92.62:41227)
0929 10:31:38.684647 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 10:31:38.684651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:31:38.684848 (+ 197us) server_negotiation.cc:407] Connection header received
0929 10:31:38.685015 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:31:38.685018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:31:38.685071 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:31:38.685168 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:31:38.686135 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.686967 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.688377 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.688674 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.688794 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:38.690383 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:31:38.690386 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:31:38.690388 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:31:38.690411 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:31:38.691901 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:31:38.692357 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:31:38.692363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:31:38.692364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:31:38.692443 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:31:38.692761 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:31:38.692764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:31:38.692766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:31:38.692963 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:31:38.693168 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 10:31:38.693526 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 10:31:38.693640 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":104,"threads_started":1}
I20260929 10:31:38.695016 4758 heartbeater.cc:347] Connected to a master server at 127.20.92.62:41227
I20260929 10:31:38.695284 4758 heartbeater.cc:464] Registering TS with master...
I20260929 10:31:38.695814 4758 heartbeater.cc:511] Master 127.20.92.62:41227 requested a full tablet report, sending...
I20260929 10:31:38.696851 4298 ts_manager.cc:194] Registered new tserver with Master: 38154f3b4e3f4ab18eeadc2377d821e7 (127.20.92.3:35189)
I20260929 10:31:38.697428 4298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.3@KRBTEST.COM'} at 127.20.92.3:39113
I20260929 10:31:38.708146 20848 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:31:38.712389 4762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:38.708902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39826 (local address 127.20.92.62:41227)
0929 10:31:38.709041 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 10:31:38.709045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:31:38.709233 (+ 188us) server_negotiation.cc:407] Connection header received
0929 10:31:38.709376 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:31:38.709379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:31:38.709485 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:31:38.709579 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:31:38.709883 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.710401 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.711284 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.711512 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.711597 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:38.711905 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:31:38.712067 (+ 162us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 10:31:38.712283 (+ 216us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39826: token expired
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 10:31:38.712491 4762 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39826: token expired
W20260929 10:31:38.712908 4227 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.20.92.62:41227)
I20260929 10:31:38.720660 4762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:38.713347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39842 (local address 127.20.92.62:41227)
0929 10:31:38.713474 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 10:31:38.713479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:31:38.713505 (+ 26us) server_negotiation.cc:407] Connection header received
0929 10:31:38.713541 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:31:38.713544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:31:38.713589 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:31:38.713681 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:31:38.714452 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.714947 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.715774 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.715999 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.716111 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:38.717508 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:31:38.717512 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:31:38.717515 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:31:38.717548 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:31:38.718934 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:31:38.719369 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:31:38.719372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:31:38.719374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:31:38.719460 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:31:38.719739 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:31:38.719743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:31:38.719745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:31:38.719967 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:31:38.720111 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 10:31:38.720361 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 10:31:38.720484 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 10:31:38.723507 4298 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:31:38.726958 4298 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:31:38.734885 4420 tablet_service.cc:1467] Processing CreateTablet for tablet 9104f8463fb046689a7ff10d4bd8d832 (DEFAULT_TABLE table=test-table [id=f9ca5f75b954487eb44ea06b2ff23914]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:31:38.735554 4420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9104f8463fb046689a7ff10d4bd8d832. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:31:38.738523 4378 tablet_bootstrap.cc:492] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13: Bootstrap starting.
I20260929 10:31:38.739535 4378 tablet_bootstrap.cc:654] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13: Neither blocks nor log segments found. Creating new log.
I20260929 10:31:38.739687 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:38.733870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:44711 (local address 127.0.0.1:38706)
0929 10:31:38.733992 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:38.734009 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:31:38.734094 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:31:38.734962 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:31:38.734964 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:31:38.734980 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:31:38.735088 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:38.735094 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.736510 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.736516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:38.738045 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:38.738057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.739263 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.739267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:38.739353 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:38.739374 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 10:31:38.739440 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 10:31:38.739548 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":30}
I20260929 10:31:38.739805 4767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:38.734039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38706 (local address 127.20.92.2:44711)
0929 10:31:38.734411 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0929 10:31:38.734427 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 10:31:38.734454 (+ 27us) server_negotiation.cc:407] Connection header received
0929 10:31:38.734538 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:31:38.734546 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:31:38.734798 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:31:38.735259 (+ 461us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:31:38.735436 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.736388 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.738194 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.739143 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.739268 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:38.739424 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 10:31:38.739480 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 10:31:38.739611 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":130,"threads_started":1}
I20260929 10:31:38.740903 4768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:38.734317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:35189 (local address 127.0.0.1:52914)
0929 10:31:38.734748 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:38.734764 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:31:38.734848 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:31:38.735575 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:31:38.735579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:31:38.735600 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:31:38.735703 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:38.735709 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.737073 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.737076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:38.738518 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:38.738530 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.740420 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.740426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:38.740564 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:38.740598 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 10:31:38.740654 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 10:31:38.740752 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":136,"thread_start_us":109,"threads_started":1}
I20260929 10:31:38.741611 4378 tablet_bootstrap.cc:492] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13: No bootstrap required, opened a new log
I20260929 10:31:38.741721 4378 ts_tablet_manager.cc:1397] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 10:31:38.741892 4769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:38.734724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52914 (local address 127.20.92.3:35189)
0929 10:31:38.735104 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0929 10:31:38.735116 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 10:31:38.735141 (+ 25us) server_negotiation.cc:407] Connection header received
0929 10:31:38.735223 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:31:38.735228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:31:38.735415 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:31:38.735570 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:31:38.735843 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.736935 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.738842 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.740256 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.741263 (+ 1007us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:38.741430 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 10:31:38.741490 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 10:31:38.741586 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":123,"threads_started":1}
I20260929 10:31:38.742261 4378 raft_consensus.cc:348] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f32320fb1e1a4b248864f8b847cadfe9" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44711 } } peers { permanent_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 35189 } } peers { permanent_uuid: "842f6d386b2849079c7ce98a3e594a13" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44141 } }
I20260929 10:31:38.742403 4378 raft_consensus.cc:374] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:31:38.742451 4378 raft_consensus.cc:729] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842f6d386b2849079c7ce98a3e594a13, State: Initialized, Role: FOLLOWER
I20260929 10:31:38.742580 4378 consensus_queue.cc:260] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f32320fb1e1a4b248864f8b847cadfe9" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44711 } } peers { permanent_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 35189 } } peers { permanent_uuid: "842f6d386b2849079c7ce98a3e594a13" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44141 } }
I20260929 10:31:38.743368 4378 ts_tablet_manager.cc:1428] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:31:38.743088 4558 tablet_service.cc:1467] Processing CreateTablet for tablet 9104f8463fb046689a7ff10d4bd8d832 (DEFAULT_TABLE table=test-table [id=f9ca5f75b954487eb44ea06b2ff23914]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:31:38.744225 4558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9104f8463fb046689a7ff10d4bd8d832. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:31:38.744827 4690 tablet_service.cc:1467] Processing CreateTablet for tablet 9104f8463fb046689a7ff10d4bd8d832 (DEFAULT_TABLE table=test-table [id=f9ca5f75b954487eb44ea06b2ff23914]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:31:38.745929 4690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9104f8463fb046689a7ff10d4bd8d832. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:31:38.752648 4771 tablet_bootstrap.cc:492] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9: Bootstrap starting.
I20260929 10:31:38.753576 4772 tablet_bootstrap.cc:492] T 9104f8463fb046689a7ff10d4bd8d832 P 38154f3b4e3f4ab18eeadc2377d821e7: Bootstrap starting.
I20260929 10:31:38.756203 4771 tablet_bootstrap.cc:654] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9: Neither blocks nor log segments found. Creating new log.
I20260929 10:31:38.756467 4772 tablet_bootstrap.cc:654] T 9104f8463fb046689a7ff10d4bd8d832 P 38154f3b4e3f4ab18eeadc2377d821e7: Neither blocks nor log segments found. Creating new log.
I20260929 10:31:38.757247 4771 log.cc:824] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9: Log is configured to *not* fsync() on all Append() calls
I20260929 10:31:38.757248 4772 log.cc:824] T 9104f8463fb046689a7ff10d4bd8d832 P 38154f3b4e3f4ab18eeadc2377d821e7: Log is configured to *not* fsync() on all Append() calls
I20260929 10:31:38.759491 4771 tablet_bootstrap.cc:492] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9: No bootstrap required, opened a new log
I20260929 10:31:38.759718 4771 ts_tablet_manager.cc:1397] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 10:31:38.760560 4772 tablet_bootstrap.cc:492] T 9104f8463fb046689a7ff10d4bd8d832 P 38154f3b4e3f4ab18eeadc2377d821e7: No bootstrap required, opened a new log
I20260929 10:31:38.760775 4772 ts_tablet_manager.cc:1397] T 9104f8463fb046689a7ff10d4bd8d832 P 38154f3b4e3f4ab18eeadc2377d821e7: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
W20260929 10:31:38.762514 4487 tablet.cc:2369] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:31:38.764369 4771 raft_consensus.cc:348] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f32320fb1e1a4b248864f8b847cadfe9" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44711 } } peers { permanent_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 35189 } } peers { permanent_uuid: "842f6d386b2849079c7ce98a3e594a13" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44141 } }
I20260929 10:31:38.764729 4771 raft_consensus.cc:374] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:31:38.764789 4771 raft_consensus.cc:729] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f32320fb1e1a4b248864f8b847cadfe9, State: Initialized, Role: FOLLOWER
I20260929 10:31:38.765029 4772 raft_consensus.cc:348] T 9104f8463fb046689a7ff10d4bd8d832 P 38154f3b4e3f4ab18eeadc2377d821e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f32320fb1e1a4b248864f8b847cadfe9" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44711 } } peers { permanent_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 35189 } } peers { permanent_uuid: "842f6d386b2849079c7ce98a3e594a13" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44141 } }
I20260929 10:31:38.765331 4772 raft_consensus.cc:374] T 9104f8463fb046689a7ff10d4bd8d832 P 38154f3b4e3f4ab18eeadc2377d821e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:31:38.765386 4772 raft_consensus.cc:729] T 9104f8463fb046689a7ff10d4bd8d832 P 38154f3b4e3f4ab18eeadc2377d821e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38154f3b4e3f4ab18eeadc2377d821e7, State: Initialized, Role: FOLLOWER
I20260929 10:31:38.765337 4771 consensus_queue.cc:260] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f32320fb1e1a4b248864f8b847cadfe9" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44711 } } peers { permanent_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 35189 } } peers { permanent_uuid: "842f6d386b2849079c7ce98a3e594a13" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44141 } }
I20260929 10:31:38.765868 4772 consensus_queue.cc:260] T 9104f8463fb046689a7ff10d4bd8d832 P 38154f3b4e3f4ab18eeadc2377d821e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f32320fb1e1a4b248864f8b847cadfe9" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44711 } } peers { permanent_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 35189 } } peers { permanent_uuid: "842f6d386b2849079c7ce98a3e594a13" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44141 } }
I20260929 10:31:38.766237 4624 heartbeater.cc:503] Master 127.20.92.62:41227 was elected leader, sending a full tablet report...
I20260929 10:31:38.766561 4771 ts_tablet_manager.cc:1428] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 10:31:38.766841 4772 ts_tablet_manager.cc:1428] T 9104f8463fb046689a7ff10d4bd8d832 P 38154f3b4e3f4ab18eeadc2377d821e7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 10:31:38.766956 4758 heartbeater.cc:503] Master 127.20.92.62:41227 was elected leader, sending a full tablet report...
I20260929 10:31:38.915321 4775 raft_consensus.cc:482] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:31:38.915589 4775 raft_consensus.cc:504] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f32320fb1e1a4b248864f8b847cadfe9" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44711 } } peers { permanent_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 35189 } } peers { permanent_uuid: "842f6d386b2849079c7ce98a3e594a13" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44141 } }
I20260929 10:31:38.916857 4775 leader_election.cc:288] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38154f3b4e3f4ab18eeadc2377d821e7 (127.20.92.3:35189), 842f6d386b2849079c7ce98a3e594a13 (127.20.92.1:44141)
I20260929 10:31:38.922394 4777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:38.917226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:59477 (local address 127.20.92.1:44141)
0929 10:31:38.917497 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0929 10:31:38.917502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:31:38.917607 (+ 105us) server_negotiation.cc:407] Connection header received
0929 10:31:38.917656 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:31:38.917659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:31:38.917827 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:31:38.917934 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:31:38.918393 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.919208 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.920977 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.921625 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.921740 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:38.921836 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 10:31:38.922175 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 10:31:38.922265 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":78,"threads_started":1}
I20260929 10:31:38.922390 4769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:38.917312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:55917 (local address 127.20.92.3:35189)
0929 10:31:38.917449 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 10:31:38.917454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:31:38.917790 (+ 336us) server_negotiation.cc:407] Connection header received
0929 10:31:38.917859 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:31:38.917871 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:31:38.917930 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:31:38.918000 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:31:38.918393 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.919193 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.920977 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.921625 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.921740 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:38.921831 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 10:31:38.922175 (+ 344us) server_negotiation.cc:299] Negotiation successful
0929 10:31:38.922256 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 10:31:38.922422 4626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:38.917150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:44141 (local address 127.20.92.2:59477)
0929 10:31:38.917275 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:38.917293 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 10:31:38.917399 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:31:38.917954 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:31:38.917956 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:31:38.917993 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:31:38.918221 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:38.918229 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.919322 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.919326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:38.920825 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:38.920836 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.921807 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.921811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:38.922014 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:38.922042 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 10:31:38.922113 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 10:31:38.922245 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":219}
I20260929 10:31:38.922431 4778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:38.917157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:35189 (local address 127.20.92.2:55917)
0929 10:31:38.917615 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:38.917626 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:31:38.917715 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:31:38.918027 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:31:38.918030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:31:38.918041 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:31:38.918220 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:38.918229 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.919322 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.919326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:38.920846 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:31:38.920852 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.921807 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.921811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:31:38.921968 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:38.921997 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 10:31:38.922095 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 10:31:38.922245 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":382,"thread_start_us":113,"threads_started":1}
I20260929 10:31:38.923172 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9104f8463fb046689a7ff10d4bd8d832" candidate_uuid: "f32320fb1e1a4b248864f8b847cadfe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842f6d386b2849079c7ce98a3e594a13" is_pre_election: true
I20260929 10:31:38.923298 4711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9104f8463fb046689a7ff10d4bd8d832" candidate_uuid: "f32320fb1e1a4b248864f8b847cadfe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" is_pre_election: true
I20260929 10:31:38.923529 4440 raft_consensus.cc:2454] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f32320fb1e1a4b248864f8b847cadfe9 in term 0.
I20260929 10:31:38.923557 4711 raft_consensus.cc:2454] T 9104f8463fb046689a7ff10d4bd8d832 P 38154f3b4e3f4ab18eeadc2377d821e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f32320fb1e1a4b248864f8b847cadfe9 in term 0.
I20260929 10:31:38.924105 4514 leader_election.cc:302] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38154f3b4e3f4ab18eeadc2377d821e7, f32320fb1e1a4b248864f8b847cadfe9; no voters:
I20260929 10:31:38.924355 4775 raft_consensus.cc:2781] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:31:38.924458 4775 raft_consensus.cc:482] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:31:38.924517 4775 raft_consensus.cc:3037] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:31:38.925730 4775 raft_consensus.cc:504] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f32320fb1e1a4b248864f8b847cadfe9" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44711 } } peers { permanent_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 35189 } } peers { permanent_uuid: "842f6d386b2849079c7ce98a3e594a13" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44141 } }
I20260929 10:31:38.926208 4775 leader_election.cc:288] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9 [CANDIDATE]: Term 1 election: Requested vote from peers 38154f3b4e3f4ab18eeadc2377d821e7 (127.20.92.3:35189), 842f6d386b2849079c7ce98a3e594a13 (127.20.92.1:44141)
I20260929 10:31:38.926556 4711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9104f8463fb046689a7ff10d4bd8d832" candidate_uuid: "f32320fb1e1a4b248864f8b847cadfe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38154f3b4e3f4ab18eeadc2377d821e7"
I20260929 10:31:38.926738 4711 raft_consensus.cc:3037] T 9104f8463fb046689a7ff10d4bd8d832 P 38154f3b4e3f4ab18eeadc2377d821e7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:31:38.926873 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9104f8463fb046689a7ff10d4bd8d832" candidate_uuid: "f32320fb1e1a4b248864f8b847cadfe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842f6d386b2849079c7ce98a3e594a13"
I20260929 10:31:38.927009 4440 raft_consensus.cc:3037] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [term 0 FOLLOWER]: Advancing to term 1
W20260929 10:31:38.928094 4759 tablet.cc:2369] T 9104f8463fb046689a7ff10d4bd8d832 P 38154f3b4e3f4ab18eeadc2377d821e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:31:38.928148 4711 raft_consensus.cc:2454] T 9104f8463fb046689a7ff10d4bd8d832 P 38154f3b4e3f4ab18eeadc2377d821e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f32320fb1e1a4b248864f8b847cadfe9 in term 1.
I20260929 10:31:38.928305 4440 raft_consensus.cc:2454] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f32320fb1e1a4b248864f8b847cadfe9 in term 1.
I20260929 10:31:38.928630 4511 leader_election.cc:302] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 842f6d386b2849079c7ce98a3e594a13, f32320fb1e1a4b248864f8b847cadfe9; no voters:
I20260929 10:31:38.928797 4775 raft_consensus.cc:2781] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:31:38.929041 4775 raft_consensus.cc:686] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9 [term 1 LEADER]: Becoming Leader. State: Replica: f32320fb1e1a4b248864f8b847cadfe9, State: Running, Role: LEADER
I20260929 10:31:38.929392 4775 consensus_queue.cc:237] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f32320fb1e1a4b248864f8b847cadfe9" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44711 } } peers { permanent_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 35189 } } peers { permanent_uuid: "842f6d386b2849079c7ce98a3e594a13" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44141 } }
I20260929 10:31:38.932710 4298 catalog_manager.cc:5495] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f32320fb1e1a4b248864f8b847cadfe9 (127.20.92.2). New cstate: current_term: 1 leader_uuid: "f32320fb1e1a4b248864f8b847cadfe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f32320fb1e1a4b248864f8b847cadfe9" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 35189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "842f6d386b2849079c7ce98a3e594a13" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44141 } health_report { overall_health: UNKNOWN } } }
W20260929 10:31:38.967070 4625 tablet.cc:2369] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:31:38.983985 4767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:38.979060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38718 (local address 127.20.92.2:44711)
0929 10:31:38.979205 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 10:31:38.979208 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:31:38.979222 (+ 14us) server_negotiation.cc:407] Connection header received
0929 10:31:38.979299 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:31:38.979301 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:31:38.979353 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:31:38.979450 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:31:38.979713 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.980284 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.981529 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:31:38.981930 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:31:38.982045 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:31:38.982517 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:31:38.983441 (+ 924us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:31:38.983610 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 10:31:38.983714 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 10:31:38.983800 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 10:31:38.992053 4440 raft_consensus.cc:1261] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [term 1 FOLLOWER]: Refusing update from remote peer f32320fb1e1a4b248864f8b847cadfe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:31:38.992077 4711 raft_consensus.cc:1261] T 9104f8463fb046689a7ff10d4bd8d832 P 38154f3b4e3f4ab18eeadc2377d821e7 [term 1 FOLLOWER]: Refusing update from remote peer f32320fb1e1a4b248864f8b847cadfe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:31:38.992892 4775 consensus_queue.cc:1035] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:31:38.993178 4779 consensus_queue.cc:1035] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "842f6d386b2849079c7ce98a3e594a13" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:31:39.000753 4783 mvcc.cc:205] Tried to move back new op lower bound from 7334616674263203840 to 7334616674017824768. Current Snapshot: MvccSnapshot[applied={T|T < 7334616674263203840}]
I20260929 10:31:39.004541 4782 mvcc.cc:205] Tried to move back new op lower bound from 7334616674263203840 to 7334616674017824768. Current Snapshot: MvccSnapshot[applied={T|T < 7334616674263203840 or (T in {7334616674263203840})}]
I20260929 10:31:39.013165 4298 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:31:39.013366 4298 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:31:39.016053 4298 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005: Sending DeleteTablet for 3 replicas of tablet 9104f8463fb046689a7ff10d4bd8d832
I20260929 10:31:39.016786 4690 tablet_service.cc:1514] Processing DeleteTablet for tablet 9104f8463fb046689a7ff10d4bd8d832 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:31:39 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:52914
I20260929 10:31:39.016785 4558 tablet_service.cc:1514] Processing DeleteTablet for tablet 9104f8463fb046689a7ff10d4bd8d832 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:31:39 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:38706
I20260929 10:31:39.017315 4792 tablet_replica.cc:330] T 9104f8463fb046689a7ff10d4bd8d832 P 38154f3b4e3f4ab18eeadc2377d821e7: stopping tablet replica
I20260929 10:31:39.017661 4792 raft_consensus.cc:2229] T 9104f8463fb046689a7ff10d4bd8d832 P 38154f3b4e3f4ab18eeadc2377d821e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:31:39.017860 4420 tablet_service.cc:1514] Processing DeleteTablet for tablet 9104f8463fb046689a7ff10d4bd8d832 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:31:39 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:41936
I20260929 10:31:39.017992 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 4336
I20260929 10:31:39.019462 4793 tablet_replica.cc:330] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9: stopping tablet replica
I20260929 10:31:39.019767 4793 raft_consensus.cc:2229] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:31:39.020150 4793 raft_consensus.cc:2258] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:31:39.018047 4792 raft_consensus.cc:2258] T 9104f8463fb046689a7ff10d4bd8d832 P 38154f3b4e3f4ab18eeadc2377d821e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:31:39.021736 4793 ts_tablet_manager.cc:1905] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:31:39.021977 4792 ts_tablet_manager.cc:1905] T 9104f8463fb046689a7ff10d4bd8d832 P 38154f3b4e3f4ab18eeadc2377d821e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:31:39.024787 4792 ts_tablet_manager.cc:1918] T 9104f8463fb046689a7ff10d4bd8d832 P 38154f3b4e3f4ab18eeadc2377d821e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:31:39.024904 4792 log.cc:1196] T 9104f8463fb046689a7ff10d4bd8d832 P 38154f3b4e3f4ab18eeadc2377d821e7: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/wal/wals/9104f8463fb046689a7ff10d4bd8d832
I20260929 10:31:39.025264 4792 ts_tablet_manager.cc:1939] T 9104f8463fb046689a7ff10d4bd8d832 P 38154f3b4e3f4ab18eeadc2377d821e7: Deleting consensus metadata
W20260929 10:31:39.025298 4282 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.92.1:44141 (error 108)
I20260929 10:31:39.025354 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 4490
I20260929 10:31:39.027346 4285 catalog_manager.cc:4841] TS 38154f3b4e3f4ab18eeadc2377d821e7 (127.20.92.3:35189): tablet 9104f8463fb046689a7ff10d4bd8d832 (table test-table [id=f9ca5f75b954487eb44ea06b2ff23914]) successfully deleted
I20260929 10:31:39.028439 4768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:39.028068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:44141 (local address 127.0.0.1:41952)
0929 10:31:39.028247 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:39.028333 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.1:44141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 10:31:39.028729 4282 catalog_manager.cc:4584] TS 842f6d386b2849079c7ce98a3e594a13 (127.20.92.1:44141): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9104f8463fb046689a7ff10d4bd8d832: Network error: Client connection negotiation failed: client connection to 127.20.92.1:44141: connect: Connection refused (error 111)
I20260929 10:31:39.034495 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 4627
I20260929 10:31:39.038153 4768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:39.034667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:44711 (local address 127.0.0.1:38732)
0929 10:31:39.037982 (+ 3315us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:39.038057 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.2:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3200}
I20260929 10:31:39.043156 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 4265
I20260929 10:31:59.051162 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:41227
--webserver_interface=127.20.92.62
--webserver_port=46847
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:41227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:31:59.178273 4796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:31:59.178591 4796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:31:59.178673 4796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:31:59.182780 4796 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 10:31:59.182873 4796 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 10:31:59.182900 4796 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 10:31:59.182926 4796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:31:59.182986 4796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:31:59.183039 4796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:31:59.183077 4796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:31:59.183110 4796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:31:59.188724 4796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:41227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:41227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=46847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:31:59.189929 4796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:31:59.190958 4796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:31:59.200034 4804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:31:59.200034 4801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:31:59.200052 4802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:31:59.200093 4796 server_base.cc:1034] running on GCE node
I20260929 10:31:59.200891 4796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:31:59.202108 4796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:31:59.203294 4796 hybrid_clock.cc:648] HybridClock initialized: now 1790677919203274 us; error 33 us; skew 500 ppm
Sep 29 10:31:59 dist-test-slave-9cj5 krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677919, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:31:59.206063 4796 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:31:59.206995 4796 webserver.cc:460] Webserver started at http://127.20.92.62:46847/ using document root <none> and password file <none>
I20260929 10:31:59.207597 4796 fs_manager.cc:359] Metadata directory not provided
I20260929 10:31:59.207659 4796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:31:59.211678 4796 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:31:59.213943 4812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:31:59.215258 4796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:31:59.215365 4796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "f0706ae6bb3b47318e36811afe109005"
format_stamp: "Formatted at 2026-09-29 10:31:16 on dist-test-slave-9cj5"
server_key: "c3b77caac26bdc748e973636cd83034d"
server_key_iv: "eab84ab7f44e19c62da80c319b5dbc23"
server_key_version: "encryptionkey@0"
I20260929 10:31:59.215911 4796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:31:59.226137 4796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:31:59.229171 4796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:31:59.229351 4796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:31:59.236707 4796 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:41227
I20260929 10:31:59.237010 4864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:41227 every 8 connection(s)
I20260929 10:31:59.237928 4796 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:31:59.238117 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 4796
I20260929 10:31:59.238639 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:44141
--local_ip_for_outbound_sockets=127.20.92.1
--tserver_master_addrs=127.20.92.62:41227
--webserver_port=41693
--webserver_interface=127.20.92.1
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:31:59.242372 4865 sys_catalog.cc:263] Verifying existing consensus state
I20260929 10:31:59.243990 4865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005: Bootstrap starting.
I20260929 10:31:59.261413 4865 log.cc:824] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005: Log is configured to *not* fsync() on all Append() calls
I20260929 10:31:59.272420 4865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:31:59.272825 4865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005: Bootstrap complete.
I20260929 10:31:59.276862 4865 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0706ae6bb3b47318e36811afe109005" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 41227 } }
I20260929 10:31:59.277278 4865 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0706ae6bb3b47318e36811afe109005, State: Initialized, Role: FOLLOWER
I20260929 10:31:59.277812 4865 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0706ae6bb3b47318e36811afe109005" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 41227 } }
I20260929 10:31:59.277954 4865 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:31:59.278038 4865 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:31:59.278154 4865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [term 2 FOLLOWER]: Advancing to term 3
I20260929 10:31:59.280040 4865 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0706ae6bb3b47318e36811afe109005" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 41227 } }
I20260929 10:31:59.280373 4865 leader_election.cc:302] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0706ae6bb3b47318e36811afe109005; no voters:
I20260929 10:31:59.280709 4865 leader_election.cc:288] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 10:31:59.280795 4869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 10:31:59.280982 4869 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [term 3 LEADER]: Becoming Leader. State: Replica: f0706ae6bb3b47318e36811afe109005, State: Running, Role: LEADER
I20260929 10:31:59.281272 4869 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0706ae6bb3b47318e36811afe109005" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 41227 } }
I20260929 10:31:59.281999 4865 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:31:59.282243 4871 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0706ae6bb3b47318e36811afe109005. Latest consensus state: current_term: 3 leader_uuid: "f0706ae6bb3b47318e36811afe109005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0706ae6bb3b47318e36811afe109005" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 41227 } } }
I20260929 10:31:59.282356 4871 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [sys.catalog]: This master's current role is: LEADER
I20260929 10:31:59.282636 4870 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f0706ae6bb3b47318e36811afe109005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0706ae6bb3b47318e36811afe109005" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 41227 } } }
I20260929 10:31:59.282733 4873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:31:59.282734 4870 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005 [sys.catalog]: This master's current role is: LEADER
I20260929 10:31:59.288054 4873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:31:59.288833 4873 catalog_manager.cc:1259] Loaded cluster ID: 10bcea7b0f30465b94c67a065b23a4c4
I20260929 10:31:59.288898 4873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:31:59.291652 4873 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:31:59.292416 4873 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005: Loaded TSK: 0
I20260929 10:31:59.292505 4873 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f0706ae6bb3b47318e36811afe109005: Loaded TSK: 1
F20260929 10:31:59.293274 4873 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f640756e9ad google::LogMessage::Fail() at ??:0
@ 0x7f640757188a google::LogMessage::SendToLog() at ??:0
@ 0x7f640756e4de google::LogMessage::Flush() at ??:0
@ 0x7f640756fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f640a23218d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f640a22af0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f6411ed37bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f6411ead99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f6411eae178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f6411eefdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f6411f0e82b std::function<>::operator()() at ??:0
@ 0x7f6411eadb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f6411eaef3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6411eab579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f6411eef640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f64097c03d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f64097c0c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f64097c23f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f64097b1512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f640adf56db start_thread at ??:0
@ 0x7f640606c61f clone at ??:0
W20260929 10:31:59.373346 4867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:31:59.373580 4867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:31:59.373624 4867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:31:59.378201 4867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:31:59.378255 4867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:31:59.378317 4867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:31:59.384639 4867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:44141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=41693
--tserver_master_addrs=127.20.92.62:41227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:31:59.385885 4867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:31:59.386899 4867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:31:59.397387 4914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:31:59.397595 4912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:31:59.397387 4911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:31:59.398710 4867 server_base.cc:1034] running on GCE node
I20260929 10:31:59.399230 4867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:31:59.399956 4867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:31:59.401212 4867 hybrid_clock.cc:648] HybridClock initialized: now 1790677919401187 us; error 52 us; skew 500 ppm
Sep 29 10:31:59 dist-test-slave-9cj5 krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677919, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:31:59.404438 4867 init.cc:377] Logged in from keytab as kudu/127.20.92.1@KRBTEST.COM (short username kudu)
I20260929 10:31:59.405535 4867 webserver.cc:460] Webserver started at http://127.20.92.1:41693/ using document root <none> and password file <none>
I20260929 10:31:59.406204 4867 fs_manager.cc:359] Metadata directory not provided
I20260929 10:31:59.406275 4867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:31:59.410566 4867 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 10:31:59.412819 4922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:31:59.414204 4867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:31:59.414332 4867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "842f6d386b2849079c7ce98a3e594a13"
format_stamp: "Formatted at 2026-09-29 10:31:16 on dist-test-slave-9cj5"
server_key: "e5da3ca21baae1b61c8b7a19a93aaf4a"
server_key_iv: "a676b4c99fd4bb9fff7214bdc75124ce"
server_key_version: "encryptionkey@0"
I20260929 10:31:59.414911 4867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:31:59.432530 4867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:31:59.436017 4867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:31:59.436221 4867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:31:59.436895 4867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:31:59.438067 4929 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 10:31:59.442211 4867 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 10:31:59.442288 4867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260929 10:31:59.442358 4867 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 10:31:59.444204 4867 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 10:31:59.444260 4867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 10:31:59.444519 4929 tablet_bootstrap.cc:492] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13: Bootstrap starting.
I20260929 10:31:59.455390 4867 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:44141
I20260929 10:31:59.455633 5037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:44141 every 8 connection(s)
I20260929 10:31:59.456666 4867 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
I20260929 10:31:59.465723 4929 log.cc:824] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13: Log is configured to *not* fsync() on all Append() calls
I20260929 10:31:59.465818 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 4867
I20260929 10:31:59.466331 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:44711
--local_ip_for_outbound_sockets=127.20.92.2
--tserver_master_addrs=127.20.92.62:41227
--webserver_port=45625
--webserver_interface=127.20.92.2
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:31:59.473390 4929 tablet_bootstrap.cc:492] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:31:59.473846 4929 tablet_bootstrap.cc:492] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13: Bootstrap complete.
I20260929 10:31:59.474921 4929 ts_tablet_manager.cc:1397] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13: Time spent bootstrapping tablet: real 0.031s user 0.018s sys 0.011s
I20260929 10:31:59.477291 4929 raft_consensus.cc:348] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f32320fb1e1a4b248864f8b847cadfe9" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44711 } } peers { permanent_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 35189 } } peers { permanent_uuid: "842f6d386b2849079c7ce98a3e594a13" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44141 } }
I20260929 10:31:59.477734 4929 raft_consensus.cc:729] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842f6d386b2849079c7ce98a3e594a13, State: Initialized, Role: FOLLOWER
I20260929 10:31:59.478246 4929 consensus_queue.cc:260] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f32320fb1e1a4b248864f8b847cadfe9" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44711 } } peers { permanent_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 35189 } } peers { permanent_uuid: "842f6d386b2849079c7ce98a3e594a13" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44141 } }
I20260929 10:31:59.479187 4929 ts_tablet_manager.cc:1428] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260929 10:31:59.554416 5040 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:31:59.459016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.1:38465)
0929 10:31:59.460080 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:59.460140 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 10:31:59.461440 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:31:59.553484 (+ 92044us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":753,"thread_start_us":156,"threads_started":1}
W20260929 10:31:59.555269 5038 heartbeater.cc:644] Failed to heartbeat to 127.20.92.62:41227 (0 consecutive failures): Network error: Failed to ping master at 127.20.92.62:41227: Client connection negotiation failed: client connection to 127.20.92.62:41227: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 10:31:59.556003 5040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:59.555755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.1:48003)
0929 10:31:59.555880 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:59.555949 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 10:31:59.556643 5040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:59.556451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.1:53033)
0929 10:31:59.556546 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:59.556590 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 10:31:59.556857 5038 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 10:31:59.596820 5042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:31:59.597071 5042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:31:59.597115 5042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:31:59.601200 5042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:31:59.601265 5042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:31:59.601357 5042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:31:59.606772 5042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:44711
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=45625
--tserver_master_addrs=127.20.92.62:41227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:31:59.607834 5042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:31:59.608825 5042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:31:59.618964 5049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:31:59.618975 5051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:31:59.619047 5048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:31:59.619074 5042 server_base.cc:1034] running on GCE node
I20260929 10:31:59.619737 5042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:31:59.620416 5042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:31:59.621640 5042 hybrid_clock.cc:648] HybridClock initialized: now 1790677919621624 us; error 28 us; skew 500 ppm
Sep 29 10:31:59 dist-test-slave-9cj5 krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677919, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:31:59.624563 5042 init.cc:377] Logged in from keytab as kudu/127.20.92.2@KRBTEST.COM (short username kudu)
I20260929 10:31:59.625430 5042 webserver.cc:460] Webserver started at http://127.20.92.2:45625/ using document root <none> and password file <none>
I20260929 10:31:59.626096 5042 fs_manager.cc:359] Metadata directory not provided
I20260929 10:31:59.626186 5042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:31:59.630061 5042 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260929 10:31:59.632542 5058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:31:59.633852 5042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:31:59.633996 5042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "f32320fb1e1a4b248864f8b847cadfe9"
format_stamp: "Formatted at 2026-09-29 10:31:17 on dist-test-slave-9cj5"
server_key: "6ba5ddf9ba44f01fd2ef8fc0f6383f79"
server_key_iv: "3901f4ac528d7646f363b96fb09cb649"
server_key_version: "encryptionkey@0"
I20260929 10:31:59.634687 5042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:31:59.651695 5042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:31:59.655531 5042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:31:59.655761 5042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:31:59.656409 5042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:31:59.657563 5065 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260929 10:31:59.661458 5065 ts_tablet_manager.cc:1852] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260929 10:31:59.661966 5065 ts_tablet_manager.cc:1905] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:31:59.666103 5065 ts_tablet_manager.cc:1918] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:31:59.666213 5065 log.cc:1196] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/wal/wals/9104f8463fb046689a7ff10d4bd8d832
I20260929 10:31:59.666604 5065 ts_tablet_manager.cc:1939] T 9104f8463fb046689a7ff10d4bd8d832 P f32320fb1e1a4b248864f8b847cadfe9: Deleting consensus metadata
I20260929 10:31:59.666951 5042 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260929 10:31:59.667042 5042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.000s
I20260929 10:31:59.667130 5042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:31:59.667171 5042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:31:59.676153 5042 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:44711
I20260929 10:31:59.676192 5173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:44711 every 8 connection(s)
I20260929 10:31:59.677045 5042 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
I20260929 10:31:59.680326 5176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:59.679091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.2:49709)
0929 10:31:59.679551 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:59.679655 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":111,"threads_started":1}
W20260929 10:31:59.680727 5174 heartbeater.cc:644] Failed to heartbeat to 127.20.92.62:41227 (0 consecutive failures): Network error: Failed to ping master at 127.20.92.62:41227: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
I20260929 10:31:59.681352 5176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:59.681141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.2:46217)
0929 10:31:59.681251 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:59.681304 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 10:31:59.682052 5176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:59.681856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.2:39433)
0929 10:31:59.681952 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:59.681996 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 10:31:59.682209 5174 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 10:31:59.685484 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 5042
I20260929 10:31:59.686039 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:35189
--local_ip_for_outbound_sockets=127.20.92.3
--tserver_master_addrs=127.20.92.62:41227
--webserver_port=44113
--webserver_interface=127.20.92.3
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:31:59.799355 5177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:31:59.799641 5177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:31:59.799688 5177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:31:59.806277 5177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:31:59.806368 5177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:31:59.806496 5177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:31:59.812875 5177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:35189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=44113
--tserver_master_addrs=127.20.92.62:41227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:31:59.814092 5177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:31:59.815109 5177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:31:59.824874 5177 server_base.cc:1034] running on GCE node
W20260929 10:31:59.824819 5182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:31:59.824827 5183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:31:59.824828 5185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:31:59.825544 5177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:31:59.826267 5177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:31:59.827435 5177 hybrid_clock.cc:648] HybridClock initialized: now 1790677919827416 us; error 34 us; skew 500 ppm
Sep 29 10:31:59 dist-test-slave-9cj5 krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677919, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:31:59.830363 5177 init.cc:377] Logged in from keytab as kudu/127.20.92.3@KRBTEST.COM (short username kudu)
I20260929 10:31:59.831207 5177 webserver.cc:460] Webserver started at http://127.20.92.3:44113/ using document root <none> and password file <none>
I20260929 10:31:59.831802 5177 fs_manager.cc:359] Metadata directory not provided
I20260929 10:31:59.831894 5177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:31:59.835762 5177 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 10:31:59.838116 5192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:31:59.839483 5177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 10:31:59.839615 5177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "38154f3b4e3f4ab18eeadc2377d821e7"
format_stamp: "Formatted at 2026-09-29 10:31:17 on dist-test-slave-9cj5"
server_key: "62d46ad117332aa631abb0449d1cc034"
server_key_iv: "d07e6904a6908e94475faea8c1a34f5b"
server_key_version: "encryptionkey@0"
I20260929 10:31:59.840161 5177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:31:59.866014 5177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:31:59.869206 5177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:31:59.869417 5177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:31:59.870048 5177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:31:59.870879 5177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:31:59.870946 5177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:31:59.871016 5177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:31:59.871055 5177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:31:59.880272 5177 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:35189
I20260929 10:31:59.880292 5306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:35189 every 8 connection(s)
I20260929 10:31:59.881300 5177 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:31:59.882939 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 5177
I20260929 10:31:59.885424 5310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:59.883576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.3:52059)
0929 10:31:59.884052 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:59.884177 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":136,"threads_started":1}
W20260929 10:31:59.886040 5307 heartbeater.cc:644] Failed to heartbeat to 127.20.92.62:41227 (0 consecutive failures): Network error: Failed to ping master at 127.20.92.62:41227: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
I20260929 10:31:59.886796 5310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:59.886523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.3:53245)
0929 10:31:59.886685 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:59.886742 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 10:31:59.887416 5310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:31:59.887225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.3:32935)
0929 10:31:59.887321 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0929 10:31:59.887370 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 10:31:59.887591 5307 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 10:32:00.558077 5311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:00.557510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.1:43889)
0929 10:32:00.557878 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:00.557964 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":84,"threads_started":1}
I20260929 10:32:00.683318 5312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:00.682898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.2:41183)
0929 10:32:00.683173 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:00.683231 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":80,"threads_started":1}
I20260929 10:32:00.888554 5313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:00.888148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.3:58677)
0929 10:32:00.888405 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:00.888455 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":88,"threads_started":1}
I20260929 10:32:01.236846 5314 raft_consensus.cc:482] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:32:01.237061 5314 raft_consensus.cc:504] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f32320fb1e1a4b248864f8b847cadfe9" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44711 } } peers { permanent_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 35189 } } peers { permanent_uuid: "842f6d386b2849079c7ce98a3e594a13" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44141 } }
I20260929 10:32:01.238297 5314 leader_election.cc:288] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f32320fb1e1a4b248864f8b847cadfe9 (127.20.92.2:44711), 38154f3b4e3f4ab18eeadc2377d821e7 (127.20.92.3:35189)
Sep 29 10:32:01 dist-test-slave-9cj5 krb5kdc[3717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677919, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for kudu/127.20.92.2@KRBTEST.COM
Sep 29 10:32:01 dist-test-slave-9cj5 krb5kdc[3717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677919, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for kudu/127.20.92.3@KRBTEST.COM
I20260929 10:32:01.252799 5317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:01.238658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:53209 (local address 127.20.92.2:44711)
0929 10:32:01.239243 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:01.239253 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:01.239284 (+ 31us) server_negotiation.cc:407] Connection header received
0929 10:32:01.240163 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:01.240172 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:01.240621 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:01.240947 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:01.242105 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:01.243059 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:01.244379 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:01.244857 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:01.245021 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:01.247837 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:01.247843 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:01.247855 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:01.247882 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:01.250046 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:01.250779 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:01.250784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:01.250788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:01.250927 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:01.251238 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:01.251241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:01.251242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:01.251657 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:01.251964 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:01.252329 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 10:32:01.252590 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":119,"threads_started":1}
I20260929 10:32:01.252803 5318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:01.238742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:53007 (local address 127.20.92.3:35189)
0929 10:32:01.239173 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:01.239197 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:01.239241 (+ 44us) server_negotiation.cc:407] Connection header received
0929 10:32:01.240172 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:01.240179 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:01.240621 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:01.240947 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:01.242123 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:01.243064 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:01.244354 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:01.244835 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:01.245007 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:01.248115 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:01.248122 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:01.248136 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:01.248171 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:01.250030 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:01.250700 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:01.250710 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:01.250719 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:01.250843 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:01.251207 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:01.251213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:01.251217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:01.251633 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:01.251737 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:01.252339 (+ 602us) server_negotiation.cc:299] Negotiation successful
0929 10:32:01.252567 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":107,"threads_started":1}
I20260929 10:32:01.252835 5315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:01.238562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:35189 (local address 127.20.92.1:53007)
0929 10:32:01.238967 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:01.238982 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:01.239092 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:01.241426 (+ 2334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:01.241428 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:01.241576 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:01.241928 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:01.241945 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:01.243260 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:01.243262 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:01.244193 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:01.244202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:01.244958 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:01.244962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:01.245195 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:01.245980 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:32:01.245997 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:01.247938 (+ 1941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:01.250151 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:01.250160 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:01.250176 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:01.250517 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:01.250936 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:01.250939 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:01.250941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:01.251111 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:01.251772 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:01.251777 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:01.252041 (+ 264us) client_negotiation.cc:769] Sending connection context
0929 10:32:01.252208 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 10:32:01.252591 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":193,"spinlock_wait_cycles":14592,"thread_start_us":93,"threads_started":1}
I20260929 10:32:01.252859 5316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:01.238503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:44711 (local address 127.20.92.1:53209)
0929 10:32:01.238867 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:01.238882 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:01.238994 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:01.241018 (+ 2024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:01.241023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:01.241394 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:01.241928 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:01.241946 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:01.243245 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:01.243247 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:01.244194 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:01.244203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:01.244972 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:01.244976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:01.245195 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:01.246014 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:32:01.246027 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:01.247578 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:01.250428 (+ 2850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:01.250431 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:01.250433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:01.250654 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:01.251050 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:01.251052 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:01.251054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:01.251134 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:01.251907 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:01.251909 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:01.252031 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 10:32:01.252200 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 10:32:01.252680 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":326,"spinlock_wait_cycles":9984,"thread_start_us":86,"threads_started":1}
W20260929 10:32:01.255421 4925 leader_election.cc:341] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f32320fb1e1a4b248864f8b847cadfe9 (127.20.92.2:44711): Not found: Tablet not found: 9104f8463fb046689a7ff10d4bd8d832
W20260929 10:32:01.255581 4926 leader_election.cc:341] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 38154f3b4e3f4ab18eeadc2377d821e7 (127.20.92.3:35189): Not found: Tablet not found: 9104f8463fb046689a7ff10d4bd8d832
I20260929 10:32:01.255659 4926 leader_election.cc:302] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 842f6d386b2849079c7ce98a3e594a13; no voters: 38154f3b4e3f4ab18eeadc2377d821e7, f32320fb1e1a4b248864f8b847cadfe9
I20260929 10:32:01.255865 5314 raft_consensus.cc:2726] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 10:32:01.559257 5315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:01.558952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.1:58185)
0929 10:32:01.559115 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:01.559172 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 10:32:01.684700 5319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:01.684286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.2:45389)
0929 10:32:01.684544 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:01.684609 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":92,"threads_started":1}
I20260929 10:32:01.889994 5320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:01.889410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.3:39363)
0929 10:32:01.889736 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:01.889873 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":102,"threads_started":1}
I20260929 10:32:02.560422 5321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:02.560018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.1:48499)
0929 10:32:02.560267 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:02.560326 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":82,"threads_started":1}
I20260929 10:32:02.686007 5322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:02.685502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.2:49367)
0929 10:32:02.685836 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:02.685897 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":79,"threads_started":1}
I20260929 10:32:02.891278 5323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:02.890832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.3:46563)
0929 10:32:02.891125 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:02.891184 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":88,"threads_started":1}
I20260929 10:32:03.289215 5324 raft_consensus.cc:482] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:32:03.289397 5324 raft_consensus.cc:504] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f32320fb1e1a4b248864f8b847cadfe9" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44711 } } peers { permanent_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 35189 } } peers { permanent_uuid: "842f6d386b2849079c7ce98a3e594a13" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44141 } }
I20260929 10:32:03.289948 5324 leader_election.cc:288] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f32320fb1e1a4b248864f8b847cadfe9 (127.20.92.2:44711), 38154f3b4e3f4ab18eeadc2377d821e7 (127.20.92.3:35189)
W20260929 10:32:03.290781 4926 leader_election.cc:341] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 38154f3b4e3f4ab18eeadc2377d821e7 (127.20.92.3:35189): Not found: Tablet not found: 9104f8463fb046689a7ff10d4bd8d832
W20260929 10:32:03.290877 4925 leader_election.cc:341] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f32320fb1e1a4b248864f8b847cadfe9 (127.20.92.2:44711): Not found: Tablet not found: 9104f8463fb046689a7ff10d4bd8d832
I20260929 10:32:03.290985 4925 leader_election.cc:302] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 842f6d386b2849079c7ce98a3e594a13; no voters: 38154f3b4e3f4ab18eeadc2377d821e7, f32320fb1e1a4b248864f8b847cadfe9
I20260929 10:32:03.291159 5324 raft_consensus.cc:2726] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 10:32:03.561728 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:03.561303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.1:56175)
0929 10:32:03.561572 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:03.561630 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":74,"threads_started":1}
I20260929 10:32:03.687348 5326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:03.686828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.2:42875)
0929 10:32:03.687136 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:03.687216 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":93,"threads_started":1}
I20260929 10:32:03.892480 5327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:03.892038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.3:60593)
0929 10:32:03.892317 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:03.892375 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":119,"threads_started":1}
I20260929 10:32:04.563117 5328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:04.562660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.1:46227)
0929 10:32:04.562958 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:04.563018 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":73,"threads_started":1}
I20260929 10:32:04.688771 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:04.688241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.2:47801)
0929 10:32:04.688533 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:04.688616 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":124,"threads_started":1}
I20260929 10:32:04.893703 5330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:04.893254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.3:39503)
0929 10:32:04.893540 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:04.893603 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":92,"threads_started":1}
I20260929 10:32:05.564355 5331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:05.563892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.1:60515)
0929 10:32:05.564198 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:05.564257 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":86,"threads_started":1}
I20260929 10:32:05.690266 5332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:05.689648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.2:56777)
0929 10:32:05.690091 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:05.690159 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":173,"threads_started":1}
I20260929 10:32:05.895079 5333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:05.894598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.3:35211)
0929 10:32:05.894907 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:05.894981 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":110,"threads_started":1}
I20260929 10:32:06.530015 5334 raft_consensus.cc:482] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:32:06.530184 5334 raft_consensus.cc:504] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f32320fb1e1a4b248864f8b847cadfe9" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44711 } } peers { permanent_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 35189 } } peers { permanent_uuid: "842f6d386b2849079c7ce98a3e594a13" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44141 } }
I20260929 10:32:06.530647 5334 leader_election.cc:288] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f32320fb1e1a4b248864f8b847cadfe9 (127.20.92.2:44711), 38154f3b4e3f4ab18eeadc2377d821e7 (127.20.92.3:35189)
W20260929 10:32:06.531337 4925 leader_election.cc:341] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f32320fb1e1a4b248864f8b847cadfe9 (127.20.92.2:44711): Not found: Tablet not found: 9104f8463fb046689a7ff10d4bd8d832
W20260929 10:32:06.531437 4926 leader_election.cc:341] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 38154f3b4e3f4ab18eeadc2377d821e7 (127.20.92.3:35189): Not found: Tablet not found: 9104f8463fb046689a7ff10d4bd8d832
I20260929 10:32:06.531541 4926 leader_election.cc:302] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 842f6d386b2849079c7ce98a3e594a13; no voters: 38154f3b4e3f4ab18eeadc2377d821e7, f32320fb1e1a4b248864f8b847cadfe9
I20260929 10:32:06.531708 5334 raft_consensus.cc:2726] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 10:32:06.565609 5335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:06.565213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.1:56117)
0929 10:32:06.565463 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:06.565514 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":91,"threads_started":1}
I20260929 10:32:06.691823 5336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:06.691320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.2:60961)
0929 10:32:06.691661 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:06.691725 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":95,"threads_started":1}
I20260929 10:32:06.896399 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:06.895901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.3:44285)
0929 10:32:06.896244 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:06.896303 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":76,"threads_started":1}
I20260929 10:32:07.566812 5338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:07.566404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.1:57969)
0929 10:32:07.566657 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:07.566714 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":74,"threads_started":1}
I20260929 10:32:07.693017 5339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:07.692611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.2:56069)
0929 10:32:07.692862 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:07.692921 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
I20260929 10:32:07.897848 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:07.897300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.3:39145)
0929 10:32:07.897604 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:07.897672 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":78,"threads_started":1}
I20260929 10:32:08.568109 5341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:08.567638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.1:33685)
0929 10:32:08.567947 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:08.568010 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":103,"threads_started":1}
I20260929 10:32:08.694144 5342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:08.693707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.2:60451)
0929 10:32:08.693999 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:08.694051 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":139,"threads_started":1}
I20260929 10:32:08.899080 5343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:08.898637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.3:50695)
0929 10:32:08.898912 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:08.898974 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":79,"threads_started":1}
I20260929 10:32:09.569447 5345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:09.569037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.1:45335)
0929 10:32:09.569301 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:09.569355 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":106,"threads_started":1}
I20260929 10:32:09.695278 5346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:09.694884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.2:44255)
0929 10:32:09.695128 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:09.695184 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
I20260929 10:32:09.900347 5347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:09.899913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.3:35941)
0929 10:32:09.900197 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:09.900254 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":73,"threads_started":1}
I20260929 10:32:10.148135 5348 raft_consensus.cc:482] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:32:10.148274 5348 raft_consensus.cc:504] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f32320fb1e1a4b248864f8b847cadfe9" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44711 } } peers { permanent_uuid: "38154f3b4e3f4ab18eeadc2377d821e7" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 35189 } } peers { permanent_uuid: "842f6d386b2849079c7ce98a3e594a13" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 44141 } }
I20260929 10:32:10.148777 5348 leader_election.cc:288] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f32320fb1e1a4b248864f8b847cadfe9 (127.20.92.2:44711), 38154f3b4e3f4ab18eeadc2377d821e7 (127.20.92.3:35189)
W20260929 10:32:10.149544 4926 leader_election.cc:341] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 38154f3b4e3f4ab18eeadc2377d821e7 (127.20.92.3:35189): Not found: Tablet not found: 9104f8463fb046689a7ff10d4bd8d832
W20260929 10:32:10.149641 4925 leader_election.cc:341] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f32320fb1e1a4b248864f8b847cadfe9 (127.20.92.2:44711): Not found: Tablet not found: 9104f8463fb046689a7ff10d4bd8d832
I20260929 10:32:10.149750 4925 leader_election.cc:302] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 842f6d386b2849079c7ce98a3e594a13; no voters: 38154f3b4e3f4ab18eeadc2377d821e7, f32320fb1e1a4b248864f8b847cadfe9
I20260929 10:32:10.149928 5348 raft_consensus.cc:2726] T 9104f8463fb046689a7ff10d4bd8d832 P 842f6d386b2849079c7ce98a3e594a13 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 10:32:10.570698 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:10.570278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.1:49979)
0929 10:32:10.570519 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:10.570577 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":79,"threads_started":1}
I20260929 10:32:10.696507 5350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:10.696070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.2:59901)
0929 10:32:10.696346 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:10.696410 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":75,"threads_started":1}
I20260929 10:32:10.901659 5351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:10.901203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.3:44571)
0929 10:32:10.901491 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:10.901551 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":91,"threads_started":1}
I20260929 10:32:11.571973 5352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:11.571510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.1:44575)
0929 10:32:11.571806 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:11.571867 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":82,"threads_started":1}
I20260929 10:32:11.697717 5353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:11.697303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.2:34121)
0929 10:32:11.697562 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:11.697616 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":83,"threads_started":1}
I20260929 10:32:11.902938 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:11.902491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.3:50065)
0929 10:32:11.902783 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:11.902844 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":76,"threads_started":1}
I20260929 10:32:12.573311 5355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:12.572837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.1:60665)
0929 10:32:12.573100 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:12.573163 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":115,"threads_started":1}
I20260929 10:32:12.698936 5356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:12.698518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.2:33083)
0929 10:32:12.698781 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:12.698839 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":77,"threads_started":1}
I20260929 10:32:12.904171 5357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:12.903736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.3:51045)
0929 10:32:12.904005 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:12.904062 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":82,"threads_started":1}
I20260929 10:32:13.574568 5358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:13.574151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.1:56983)
0929 10:32:13.574409 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:13.574468 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":83,"threads_started":1}
I20260929 10:32:13.700055 5359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:13.699645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.2:54975)
0929 10:32:13.699899 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:13.699959 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":75,"threads_started":1}
I20260929 10:32:13.905501 5360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:13.905069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.3:46379)
0929 10:32:13.905342 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:13.905404 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":80,"threads_started":1}
I20260929 10:32:14.575851 5361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:14.575454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.1:54437)
0929 10:32:14.575710 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:14.575759 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":72,"threads_started":1}
I20260929 10:32:14.701336 5362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:14.700931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41227 (local address 127.20.92.2:55463)
0929 10:32:14.701189 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:14.701244 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.62:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":99,"threads_started":1}
I20260929 10:32:14.887970 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 4867
I20260929 10:32:14.893908 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 5042
I20260929 10:32:14.899358 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 5177
2026-09-29T10:32:14Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59660 ms)
[----------] 30 tests from SecurityITest (255605 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:32:14 dist-test-slave-9cj5 krb5kdc[5365](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:32:14 dist-test-slave-9cj5 krb5kdc[5365](info): set up 2 sockets
Sep 29 10:32:14 dist-test-slave-9cj5 krb5kdc[5365](info): commencing operation
krb5kdc: starting...
W20260929 10:32:15.962517 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:32:16 dist-test-slave-9cj5 krb5kdc[5365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:32:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:32:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
I20260929 10:32:16.093577 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:38591
--webserver_interface=127.20.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:37195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:38591
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:32:16.207676 5378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:16.208005 5378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:16.208088 5378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:16.212256 5378 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:32:16.212353 5378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:16.212391 5378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:16.212451 5378 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:32:16.212491 5378 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:32:16.217916 5378 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:37195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:38591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:38591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:16.219003 5378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:16.220036 5378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:32:16.228853 5378 server_base.cc:1034] running on GCE node
W20260929 10:32:16.228972 5383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:16.228972 5386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:16.229118 5384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:16.229986 5378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:16.231159 5378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:16.232375 5378 hybrid_clock.cc:648] HybridClock initialized: now 1790677936232355 us; error 33 us; skew 500 ppm
Sep 29 10:32:16 dist-test-slave-9cj5 krb5kdc[5365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677936, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:16.235481 5378 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:32:16.236584 5378 webserver.cc:460] Webserver started at http://127.20.92.62:43207/ using document root <none> and password file <none>
I20260929 10:32:16.237221 5378 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:16.237323 5378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:16.237576 5378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:16.239941 5378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "8a31a9d437934b7ba634ec1d3dd03548"
format_stamp: "Formatted at 2026-09-29 10:32:16 on dist-test-slave-9cj5"
server_key: "e78a5e3736117e672d30e73e81cc1e83"
server_key_iv: "1dd0f8e9fa24fa7ea1513057fd3c4085"
server_key_version: "encryptionkey@0"
I20260929 10:32:16.240566 5378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "8a31a9d437934b7ba634ec1d3dd03548"
format_stamp: "Formatted at 2026-09-29 10:32:16 on dist-test-slave-9cj5"
server_key: "e78a5e3736117e672d30e73e81cc1e83"
server_key_iv: "1dd0f8e9fa24fa7ea1513057fd3c4085"
server_key_version: "encryptionkey@0"
I20260929 10:32:16.244704 5378 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 10:32:16.247567 5393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:16.248951 5378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 10:32:16.249116 5378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "8a31a9d437934b7ba634ec1d3dd03548"
format_stamp: "Formatted at 2026-09-29 10:32:16 on dist-test-slave-9cj5"
server_key: "e78a5e3736117e672d30e73e81cc1e83"
server_key_iv: "1dd0f8e9fa24fa7ea1513057fd3c4085"
server_key_version: "encryptionkey@0"
I20260929 10:32:16.249251 5378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:16.265213 5378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:16.268858 5378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:16.269090 5378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:16.276957 5378 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:38591
I20260929 10:32:16.276976 5445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:38591 every 8 connection(s)
I20260929 10:32:16.278029 5378 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:32:16.280223 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 5378
I20260929 10:32:16.280380 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:32:16.280725 20848 external_mini_cluster.cc:1428] Setting key cda0741d1c3b544d071acd14abe634a9
I20260929 10:32:16.281015 5446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:16.288352 5446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a31a9d437934b7ba634ec1d3dd03548: Bootstrap starting.
Sep 29 10:32:16 dist-test-slave-9cj5 krb5kdc[5365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:32:16.290755 5446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a31a9d437934b7ba634ec1d3dd03548: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:16.291685 5446 log.cc:824] T 00000000000000000000000000000000 P 8a31a9d437934b7ba634ec1d3dd03548: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:16.294160 5446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a31a9d437934b7ba634ec1d3dd03548: No bootstrap required, opened a new log
I20260929 10:32:16.296208 5449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:16.282222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42772 (local address 127.20.92.62:38591)
0929 10:32:16.282677 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:16.282696 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:16.282737 (+ 41us) server_negotiation.cc:407] Connection header received
0929 10:32:16.283469 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:16.283491 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:16.283904 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:16.284274 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:16.284993 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:16.286510 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:16.287544 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:16.288215 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:16.288375 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:16.290906 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:16.290912 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:16.290929 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:16.290967 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:16.292970 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:16.293446 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:16.293452 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:16.293458 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:16.293617 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:16.293980 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:16.293983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:16.293985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:16.294488 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:16.294677 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:16.294965 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 10:32:16.295270 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":142,"threads_started":1}
I20260929 10:32:16.298619 5446 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a31a9d437934b7ba634ec1d3dd03548 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a31a9d437934b7ba634ec1d3dd03548" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38591 } }
I20260929 10:32:16.298841 5446 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a31a9d437934b7ba634ec1d3dd03548 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:16.298877 5446 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a31a9d437934b7ba634ec1d3dd03548 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a31a9d437934b7ba634ec1d3dd03548, State: Initialized, Role: FOLLOWER
I20260929 10:32:16.299271 5446 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a31a9d437934b7ba634ec1d3dd03548 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a31a9d437934b7ba634ec1d3dd03548" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38591 } }
I20260929 10:32:16.299389 5446 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a31a9d437934b7ba634ec1d3dd03548 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:32:16.299433 5446 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a31a9d437934b7ba634ec1d3dd03548 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:32:16.299511 5446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a31a9d437934b7ba634ec1d3dd03548 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:16.300607 5446 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a31a9d437934b7ba634ec1d3dd03548 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a31a9d437934b7ba634ec1d3dd03548" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38591 } }
I20260929 10:32:16.300891 5446 leader_election.cc:302] T 00000000000000000000000000000000 P 8a31a9d437934b7ba634ec1d3dd03548 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a31a9d437934b7ba634ec1d3dd03548; no voters:
I20260929 10:32:16.301142 5446 leader_election.cc:288] T 00000000000000000000000000000000 P 8a31a9d437934b7ba634ec1d3dd03548 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:32:16.301280 5451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a31a9d437934b7ba634ec1d3dd03548 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:32:16.301530 5451 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a31a9d437934b7ba634ec1d3dd03548 [term 1 LEADER]: Becoming Leader. State: Replica: 8a31a9d437934b7ba634ec1d3dd03548, State: Running, Role: LEADER
I20260929 10:32:16.301908 5451 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a31a9d437934b7ba634ec1d3dd03548 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a31a9d437934b7ba634ec1d3dd03548" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38591 } }
I20260929 10:32:16.302428 5446 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a31a9d437934b7ba634ec1d3dd03548 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:32:16.303656 5453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a31a9d437934b7ba634ec1d3dd03548 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a31a9d437934b7ba634ec1d3dd03548. Latest consensus state: current_term: 1 leader_uuid: "8a31a9d437934b7ba634ec1d3dd03548" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a31a9d437934b7ba634ec1d3dd03548" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38591 } } }
I20260929 10:32:16.303637 5452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a31a9d437934b7ba634ec1d3dd03548 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a31a9d437934b7ba634ec1d3dd03548" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a31a9d437934b7ba634ec1d3dd03548" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38591 } } }
I20260929 10:32:16.303783 5453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a31a9d437934b7ba634ec1d3dd03548 [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:16.303783 5452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a31a9d437934b7ba634ec1d3dd03548 [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:16.304706 5461 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:32:16.307399 5461 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:32:16.312785 5461 catalog_manager.cc:1347] Generated new cluster ID: bbb5b7918c494203a64d92e305aed2bb
I20260929 10:32:16.312855 5461 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:32:16.338408 5461 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:32:16.339768 5461 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:32:16.346690 5461 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a31a9d437934b7ba634ec1d3dd03548: Generated new TSK 0
I20260929 10:32:16.347579 5461 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
I20260929 10:32:16.386109 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:0
--local_ip_for_outbound_sockets=127.20.92.1
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:38591
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:37195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:32:16.499575 5472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:16.499867 5472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:16.499915 5472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:16.504293 5472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:16.504364 5472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:16.504477 5472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:32:16.509976 5472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:37195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:38591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:16.511087 5472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:16.512188 5472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:16.522231 5480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:16.522372 5472 server_base.cc:1034] running on GCE node
W20260929 10:32:16.522202 5478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:16.522590 5477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:16.523061 5472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:16.523808 5472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:16.525002 5472 hybrid_clock.cc:648] HybridClock initialized: now 1790677936524984 us; error 30 us; skew 500 ppm
Sep 29 10:32:16 dist-test-slave-9cj5 krb5kdc[5365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677936, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:16.528236 5472 init.cc:377] Logged in from keytab as kudu/127.20.92.1@KRBTEST.COM (short username kudu)
I20260929 10:32:16.529114 5472 webserver.cc:460] Webserver started at http://127.20.92.1:38801/ using document root <none> and password file <none>
I20260929 10:32:16.529649 5472 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:16.529706 5472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:16.529951 5472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:16.532002 5472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-0/data/instance:
uuid: "791e52d5e3b84ee3a8c9025dde9726ce"
format_stamp: "Formatted at 2026-09-29 10:32:16 on dist-test-slave-9cj5"
server_key: "93fbe4a79e212ca4687ea9508daeb0eb"
server_key_iv: "303f4fc13b4517600e0db85dd611c9d4"
server_key_version: "encryptionkey@0"
I20260929 10:32:16.532505 5472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance:
uuid: "791e52d5e3b84ee3a8c9025dde9726ce"
format_stamp: "Formatted at 2026-09-29 10:32:16 on dist-test-slave-9cj5"
server_key: "93fbe4a79e212ca4687ea9508daeb0eb"
server_key_iv: "303f4fc13b4517600e0db85dd611c9d4"
server_key_version: "encryptionkey@0"
I20260929 10:32:16.536339 5472 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:32:16.538889 5487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:16.540280 5472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:32:16.540423 5472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "791e52d5e3b84ee3a8c9025dde9726ce"
format_stamp: "Formatted at 2026-09-29 10:32:16 on dist-test-slave-9cj5"
server_key: "93fbe4a79e212ca4687ea9508daeb0eb"
server_key_iv: "303f4fc13b4517600e0db85dd611c9d4"
server_key_version: "encryptionkey@0"
I20260929 10:32:16.540544 5472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:16.569033 5472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:16.572319 5472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:16.572547 5472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:16.573163 5472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:32:16.574060 5472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:32:16.574131 5472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:16.574205 5472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:32:16.574257 5472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:16.582901 5472 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:45927
I20260929 10:32:16.582922 5601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:45927 every 8 connection(s)
I20260929 10:32:16.583812 5472 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
I20260929 10:32:16.593379 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 5472
I20260929 10:32:16.593542 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance
I20260929 10:32:16.593967 20848 external_mini_cluster.cc:1428] Setting key b9d1ce8db40b068e4254837aa7849ac1
Sep 29 10:32:16 dist-test-slave-9cj5 krb5kdc[5365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677936, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:32:16.599346 5449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:16.585683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:44321 (local address 127.20.92.62:38591)
0929 10:32:16.585881 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:16.585885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:16.587351 (+ 1466us) server_negotiation.cc:407] Connection header received
0929 10:32:16.588283 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:16.588286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:16.588339 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:16.588440 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:16.589661 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:16.590281 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:16.591459 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:16.591748 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:16.591843 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:16.594973 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:16.594977 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:16.594979 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:16.595013 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:16.596793 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:16.597483 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:16.597489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:16.597491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:16.597585 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:16.598057 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:16.598061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:16.598063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:16.598275 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:16.598397 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:16.599002 (+ 605us) server_negotiation.cc:299] Negotiation successful
0929 10:32:16.599155 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 10:32:16.600214 5604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:16.586137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38591 (local address 127.20.92.1:44321)
0929 10:32:16.587219 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:16.587261 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:16.588061 (+ 800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:16.588622 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:16.588629 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:16.588992 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:16.589457 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:16.589470 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:16.590404 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:16.590406 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:16.591304 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:16.591313 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:16.591927 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:16.591932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:16.592092 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:16.592806 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:32:16.592824 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:16.594788 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:16.596965 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:16.596970 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:16.596985 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:16.597357 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:16.597700 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:16.597705 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:16.597708 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:16.597920 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:16.598395 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:16.598403 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:16.598721 (+ 318us) client_negotiation.cc:769] Sending connection context
0929 10:32:16.598928 (+ 207us) client_negotiation.cc:240] Negotiation successful
0929 10:32:16.599308 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":879,"thread_start_us":551,"threads_started":1}
I20260929 10:32:16.601675 5602 heartbeater.cc:347] Connected to a master server at 127.20.92.62:38591
I20260929 10:32:16.602015 5602 heartbeater.cc:464] Registering TS with master...
I20260929 10:32:16.602685 5602 heartbeater.cc:511] Master 127.20.92.62:38591 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.92.2@KRBTEST.COM; defaulting to no policy
I20260929 10:32:16.604600 5410 ts_manager.cc:194] Registered new tserver with Master: 791e52d5e3b84ee3a8c9025dde9726ce (127.20.92.1:45927)
I20260929 10:32:16.605917 5410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.1@KRBTEST.COM'} at 127.20.92.1:44321
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
I20260929 10:32:16.626416 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:0
--local_ip_for_outbound_sockets=127.20.92.2
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:38591
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:37195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:32:16.739593 5607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:16.739878 5607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:16.739962 5607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:16.744122 5607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:16.744184 5607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:16.744242 5607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:32:16.749843 5607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:37195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:38591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:16.750891 5607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:16.751888 5607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:32:16.762066 5607 server_base.cc:1034] running on GCE node
W20260929 10:32:16.762106 5616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:16.762214 5614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:16.762331 5613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:16.762792 5607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:16.763428 5607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:16.764631 5607 hybrid_clock.cc:648] HybridClock initialized: now 1790677936764593 us; error 48 us; skew 500 ppm
Sep 29 10:32:16 dist-test-slave-9cj5 krb5kdc[5365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677936, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:16.767413 5607 init.cc:377] Logged in from keytab as kudu/127.20.92.2@KRBTEST.COM (short username kudu)
I20260929 10:32:16.768318 5607 webserver.cc:460] Webserver started at http://127.20.92.2:42717/ using document root <none> and password file <none>
I20260929 10:32:16.768888 5607 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:16.768980 5607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:16.769222 5607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:16.771320 5607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-1/data/instance:
uuid: "cf9b3ba5ac8f459992fae40f75348e84"
format_stamp: "Formatted at 2026-09-29 10:32:16 on dist-test-slave-9cj5"
server_key: "74cb6ffbfdd7571177c1a93041e9faf4"
server_key_iv: "2ba88fc6dbd5ab60e224b8d9e88b7a9a"
server_key_version: "encryptionkey@0"
I20260929 10:32:16.771950 5607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance:
uuid: "cf9b3ba5ac8f459992fae40f75348e84"
format_stamp: "Formatted at 2026-09-29 10:32:16 on dist-test-slave-9cj5"
server_key: "74cb6ffbfdd7571177c1a93041e9faf4"
server_key_iv: "2ba88fc6dbd5ab60e224b8d9e88b7a9a"
server_key_version: "encryptionkey@0"
I20260929 10:32:16.775615 5607 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 10:32:16.778054 5623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:16.779363 5607 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:32:16.779511 5607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "cf9b3ba5ac8f459992fae40f75348e84"
format_stamp: "Formatted at 2026-09-29 10:32:16 on dist-test-slave-9cj5"
server_key: "74cb6ffbfdd7571177c1a93041e9faf4"
server_key_iv: "2ba88fc6dbd5ab60e224b8d9e88b7a9a"
server_key_version: "encryptionkey@0"
I20260929 10:32:16.779630 5607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:16.791575 5607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:16.794729 5607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:16.794945 5607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:16.795554 5607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:32:16.796360 5607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:32:16.796406 5607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:16.796476 5607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:32:16.796514 5607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:16.805377 5607 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:33223
I20260929 10:32:16.805611 5736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:33223 every 8 connection(s)
I20260929 10:32:16.806358 5607 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
I20260929 10:32:16.813587 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 5607
I20260929 10:32:16.813715 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance
I20260929 10:32:16.814020 20848 external_mini_cluster.cc:1428] Setting key 5ee145d1d7fd7d3b5deb831a6bc3d0de
Sep 29 10:32:16 dist-test-slave-9cj5 krb5kdc[5365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677936, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:32:16.821800 5449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:16.808315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:49471 (local address 127.20.92.62:38591)
0929 10:32:16.808535 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:16.808539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:16.809305 (+ 766us) server_negotiation.cc:407] Connection header received
0929 10:32:16.811016 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:16.811019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:16.811075 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:16.811182 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:16.812318 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:16.812851 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:16.814390 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:16.814688 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:16.814817 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:16.817672 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:16.817675 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:16.817678 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:16.817702 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:16.819161 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:16.819704 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:16.819712 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:16.819714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:16.819784 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:16.820248 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:16.820255 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:16.820259 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:16.820586 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:16.820720 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:16.821423 (+ 703us) server_negotiation.cc:299] Negotiation successful
0929 10:32:16.821556 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 10:32:16.822613 5739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:16.808687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38591 (local address 127.20.92.2:49471)
0929 10:32:16.809155 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:16.809195 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:16.810810 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:16.811357 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:16.811364 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:16.811709 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:16.812173 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:16.812186 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:16.812976 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:16.812979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:16.814251 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:16.814261 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:16.814852 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:16.814856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:16.815030 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:16.815675 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:32:16.815692 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:16.817492 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:16.819282 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:16.819288 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:16.819303 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:16.819587 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:16.819880 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:16.819884 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:16.819886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:16.820068 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:16.820729 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:16.820757 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:16.821135 (+ 378us) client_negotiation.cc:769] Sending connection context
0929 10:32:16.821337 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 10:32:16.821670 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":120,"threads_started":1}
WARNING: no policy specified for kudu/127.20.92.3@KRBTEST.COM; defaulting to no policy
I20260929 10:32:16.823824 5737 heartbeater.cc:347] Connected to a master server at 127.20.92.62:38591
I20260929 10:32:16.824074 5737 heartbeater.cc:464] Registering TS with master...
I20260929 10:32:16.824589 5737 heartbeater.cc:511] Master 127.20.92.62:38591 requested a full tablet report, sending...
I20260929 10:32:16.825662 5410 ts_manager.cc:194] Registered new tserver with Master: cf9b3ba5ac8f459992fae40f75348e84 (127.20.92.2:33223)
I20260929 10:32:16.826335 5410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.2@KRBTEST.COM'} at 127.20.92.2:49471
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
I20260929 10:32:16.842067 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:0
--local_ip_for_outbound_sockets=127.20.92.3
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:38591
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:37195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:32:16.951715 5743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:16.952015 5743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:16.952095 5743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:16.956260 5743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:16.956351 5743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:16.956440 5743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:32:16.961977 5743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:37195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:38591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:16.963017 5743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:16.964022 5743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:16.973996 5749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:16.973996 5751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:16.974025 5748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:16.974241 5743 server_base.cc:1034] running on GCE node
I20260929 10:32:16.974779 5743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:16.975641 5743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:16.976790 5743 hybrid_clock.cc:648] HybridClock initialized: now 1790677936976753 us; error 55 us; skew 500 ppm
Sep 29 10:32:16 dist-test-slave-9cj5 krb5kdc[5365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677936, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:16.980629 5743 init.cc:377] Logged in from keytab as kudu/127.20.92.3@KRBTEST.COM (short username kudu)
I20260929 10:32:16.981993 5743 webserver.cc:460] Webserver started at http://127.20.92.3:46257/ using document root <none> and password file <none>
I20260929 10:32:16.982685 5743 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:16.982769 5743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:16.982996 5743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:16.985378 5743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-2/data/instance:
uuid: "8b7713f10ebb487aa086154a0edd1349"
format_stamp: "Formatted at 2026-09-29 10:32:16 on dist-test-slave-9cj5"
server_key: "5c282ff5556938697701cae6f6ec786a"
server_key_iv: "536afdd54a3da3e0c79498e10f6b0a34"
server_key_version: "encryptionkey@0"
I20260929 10:32:16.986004 5743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance:
uuid: "8b7713f10ebb487aa086154a0edd1349"
format_stamp: "Formatted at 2026-09-29 10:32:16 on dist-test-slave-9cj5"
server_key: "5c282ff5556938697701cae6f6ec786a"
server_key_iv: "536afdd54a3da3e0c79498e10f6b0a34"
server_key_version: "encryptionkey@0"
I20260929 10:32:16.989943 5743 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 10:32:16.992412 5758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:16.993667 5743 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:32:16.993836 5743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "8b7713f10ebb487aa086154a0edd1349"
format_stamp: "Formatted at 2026-09-29 10:32:16 on dist-test-slave-9cj5"
server_key: "5c282ff5556938697701cae6f6ec786a"
server_key_iv: "536afdd54a3da3e0c79498e10f6b0a34"
server_key_version: "encryptionkey@0"
I20260929 10:32:16.993961 5743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:17.012499 5743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:17.017194 5743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:17.017407 5743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:17.018131 5743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:32:17.018977 5743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:32:17.019049 5743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:17.019114 5743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:32:17.019167 5743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:17.028364 5743 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:41767
I20260929 10:32:17.028389 5871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:41767 every 8 connection(s)
I20260929 10:32:17.029281 5743 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:32:17.038812 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 5743
Sep 29 10:32:17 dist-test-slave-9cj5 krb5kdc[5365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677936, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:32:17.038971 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance
I20260929 10:32:17.039304 20848 external_mini_cluster.cc:1428] Setting key 760205df7f4312435d2be0ccdcc65240
I20260929 10:32:17.043629 5449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:17.031157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:38893 (local address 127.20.92.62:38591)
0929 10:32:17.031332 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:17.031337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:17.032125 (+ 788us) server_negotiation.cc:407] Connection header received
0929 10:32:17.033556 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:17.033559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:17.033615 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:17.033710 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:17.034945 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.035474 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.036621 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.036876 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.036986 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:17.039774 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:17.039779 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:17.039782 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:17.039817 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:17.041683 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:17.042248 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:17.042251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:17.042253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:17.042335 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:17.042644 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:17.042646 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:17.042647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:17.042832 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:17.042914 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:17.043396 (+ 482us) server_negotiation.cc:299] Negotiation successful
0929 10:32:17.043502 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 10:32:17.044533 5874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:17.031537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38591 (local address 127.20.92.3:38893)
0929 10:32:17.031961 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:17.032001 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:17.033265 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:17.033938 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:17.033946 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:17.034323 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:17.034787 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:17.034800 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.035608 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.035611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:17.036484 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:17.036493 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.037028 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.037031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:17.037175 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:17.037850 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:32:17.037867 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:17.039542 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:17.041855 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:17.041860 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:17.041873 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:17.042144 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:17.042419 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:17.042421 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:17.042422 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:17.042550 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:17.042921 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:17.042926 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:17.043169 (+ 243us) client_negotiation.cc:769] Sending connection context
0929 10:32:17.043346 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 10:32:17.043600 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":103,"threads_started":1}
I20260929 10:32:17.045634 5872 heartbeater.cc:347] Connected to a master server at 127.20.92.62:38591
I20260929 10:32:17.045900 5872 heartbeater.cc:464] Registering TS with master...
I20260929 10:32:17.046340 5872 heartbeater.cc:511] Master 127.20.92.62:38591 requested a full tablet report, sending...
I20260929 10:32:17.047307 5410 ts_manager.cc:194] Registered new tserver with Master: 8b7713f10ebb487aa086154a0edd1349 (127.20.92.3:41767)
I20260929 10:32:17.047847 5410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.3@KRBTEST.COM'} at 127.20.92.3:38893
I20260929 10:32:17.053598 20848 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:32:17.063491 5449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:17.055802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42776 (local address 127.20.92.62:38591)
0929 10:32:17.055959 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:17.055964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:17.056093 (+ 129us) server_negotiation.cc:407] Connection header received
0929 10:32:17.056172 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:17.056174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:17.056214 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:17.056282 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:17.056986 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.057514 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.058418 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.058684 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.058786 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:17.060054 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:17.060057 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:17.060060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:17.060084 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:17.061702 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:17.062175 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:17.062178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:17.062179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:17.062269 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:17.062538 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:17.062575 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:17.062576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:17.062848 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:17.062967 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:17.063245 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 10:32:17.063348 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 10:32:17.066457 5410 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:32:17.069468 5410 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:32:17.084782 5885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:17.078515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55416 (local address 127.20.92.2:33223)
0929 10:32:17.079602 (+ 1087us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:17.079615 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:17.079641 (+ 26us) server_negotiation.cc:407] Connection header received
0929 10:32:17.079727 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:17.079733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:17.079970 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:17.080138 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:32:17.080475 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.081374 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.083057 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.084059 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.084206 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:17.084355 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:17.084495 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 10:32:17.084603 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":971,"thread_start_us":176,"threads_started":1}
I20260929 10:32:17.084776 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:17.078321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:33223 (local address 127.0.0.1:55416)
0929 10:32:17.078777 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:17.078838 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:17.079000 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:17.080153 (+ 1153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:17.080161 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:17.080189 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:32:17.080337 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:17.080347 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.081500 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.081502 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:17.082948 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:17.082961 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.084230 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.084234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:17.084336 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:17.084354 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 10:32:17.084462 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 10:32:17.084615 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":87,"thread_start_us":118,"threads_started":1}
I20260929 10:32:17.084883 5889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:17.079299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38652 (local address 127.20.92.3:41767)
0929 10:32:17.079743 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:17.079757 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:17.079782 (+ 25us) server_negotiation.cc:407] Connection header received
0929 10:32:17.079860 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:17.079868 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:17.080057 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:17.080242 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:32:17.080521 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.081844 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.083260 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.083953 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.084083 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:17.084256 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:17.084495 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 10:32:17.084589 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":160,"threads_started":1}
I20260929 10:32:17.084781 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:17.079066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:41767 (local address 127.0.0.1:38652)
0929 10:32:17.079633 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:17.079646 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:17.079725 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:17.080236 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:17.080240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:17.080258 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:32:17.080372 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:17.080379 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.081985 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.081988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:17.083151 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:17.083158 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.084162 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.084166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:17.084280 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:17.084329 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 10:32:17.084451 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 10:32:17.084611 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":30,"thread_start_us":86,"threads_started":1}
I20260929 10:32:17.086293 5886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:17.078760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:45927 (local address 127.0.0.1:51968)
0929 10:32:17.079327 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:17.079342 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:17.079436 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:17.080958 (+ 1522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:17.080962 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:17.080980 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:32:17.081068 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:17.081073 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.082570 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.082574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:17.083895 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:17.083907 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.085864 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.085870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:17.085973 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:17.085998 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 10:32:17.086067 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 10:32:17.086139 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":118,"threads_started":1}
I20260929 10:32:17.086917 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:17.079057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51968 (local address 127.20.92.1:45927)
0929 10:32:17.080508 (+ 1451us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:17.080523 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:17.080546 (+ 23us) server_negotiation.cc:407] Connection header received
0929 10:32:17.080621 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:17.080626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:17.080809 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:17.080962 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:32:17.081182 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.082414 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.084839 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.085681 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.086559 (+ 878us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:17.086663 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:17.086705 (+ 42us) server_negotiation.cc:299] Negotiation successful
0929 10:32:17.086793 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1331,"thread_start_us":97,"threads_started":1}
I20260929 10:32:17.088413 5671 tablet_service.cc:1467] Processing CreateTablet for tablet 77fa0564fd3f46fcabecbffc015440e2 (DEFAULT_TABLE table=test-table [id=d3f3d6c0220e40ca928cf81c8b1ed89c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:32:17.088413 5806 tablet_service.cc:1467] Processing CreateTablet for tablet 77fa0564fd3f46fcabecbffc015440e2 (DEFAULT_TABLE table=test-table [id=d3f3d6c0220e40ca928cf81c8b1ed89c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:32:17.089543 5806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77fa0564fd3f46fcabecbffc015440e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:17.089538 5671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77fa0564fd3f46fcabecbffc015440e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:17.089367 5536 tablet_service.cc:1467] Processing CreateTablet for tablet 77fa0564fd3f46fcabecbffc015440e2 (DEFAULT_TABLE table=test-table [id=d3f3d6c0220e40ca928cf81c8b1ed89c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:32:17.090430 5536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77fa0564fd3f46fcabecbffc015440e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:17.096644 5890 tablet_bootstrap.cc:492] T 77fa0564fd3f46fcabecbffc015440e2 P 791e52d5e3b84ee3a8c9025dde9726ce: Bootstrap starting.
I20260929 10:32:17.097730 5891 tablet_bootstrap.cc:492] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349: Bootstrap starting.
I20260929 10:32:17.099498 5890 tablet_bootstrap.cc:654] T 77fa0564fd3f46fcabecbffc015440e2 P 791e52d5e3b84ee3a8c9025dde9726ce: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:17.100129 5891 tablet_bootstrap.cc:654] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:17.101074 5890 log.cc:824] T 77fa0564fd3f46fcabecbffc015440e2 P 791e52d5e3b84ee3a8c9025dde9726ce: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:17.101480 5891 log.cc:824] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:17.101895 5892 tablet_bootstrap.cc:492] T 77fa0564fd3f46fcabecbffc015440e2 P cf9b3ba5ac8f459992fae40f75348e84: Bootstrap starting.
I20260929 10:32:17.103161 5890 tablet_bootstrap.cc:492] T 77fa0564fd3f46fcabecbffc015440e2 P 791e52d5e3b84ee3a8c9025dde9726ce: No bootstrap required, opened a new log
I20260929 10:32:17.103370 5890 ts_tablet_manager.cc:1397] T 77fa0564fd3f46fcabecbffc015440e2 P 791e52d5e3b84ee3a8c9025dde9726ce: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 10:32:17.103737 5891 tablet_bootstrap.cc:492] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349: No bootstrap required, opened a new log
I20260929 10:32:17.103961 5891 ts_tablet_manager.cc:1397] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260929 10:32:17.104326 5892 tablet_bootstrap.cc:654] T 77fa0564fd3f46fcabecbffc015440e2 P cf9b3ba5ac8f459992fae40f75348e84: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:17.105701 5892 log.cc:824] T 77fa0564fd3f46fcabecbffc015440e2 P cf9b3ba5ac8f459992fae40f75348e84: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:17.106684 5890 raft_consensus.cc:348] T 77fa0564fd3f46fcabecbffc015440e2 P 791e52d5e3b84ee3a8c9025dde9726ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b3ba5ac8f459992fae40f75348e84" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 33223 } } peers { permanent_uuid: "8b7713f10ebb487aa086154a0edd1349" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 41767 } } peers { permanent_uuid: "791e52d5e3b84ee3a8c9025dde9726ce" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 45927 } }
I20260929 10:32:17.106931 5890 raft_consensus.cc:374] T 77fa0564fd3f46fcabecbffc015440e2 P 791e52d5e3b84ee3a8c9025dde9726ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:17.106982 5890 raft_consensus.cc:729] T 77fa0564fd3f46fcabecbffc015440e2 P 791e52d5e3b84ee3a8c9025dde9726ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 791e52d5e3b84ee3a8c9025dde9726ce, State: Initialized, Role: FOLLOWER
I20260929 10:32:17.107497 5890 consensus_queue.cc:260] T 77fa0564fd3f46fcabecbffc015440e2 P 791e52d5e3b84ee3a8c9025dde9726ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b3ba5ac8f459992fae40f75348e84" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 33223 } } peers { permanent_uuid: "8b7713f10ebb487aa086154a0edd1349" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 41767 } } peers { permanent_uuid: "791e52d5e3b84ee3a8c9025dde9726ce" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 45927 } }
I20260929 10:32:17.107946 5892 tablet_bootstrap.cc:492] T 77fa0564fd3f46fcabecbffc015440e2 P cf9b3ba5ac8f459992fae40f75348e84: No bootstrap required, opened a new log
I20260929 10:32:17.108126 5892 ts_tablet_manager.cc:1397] T 77fa0564fd3f46fcabecbffc015440e2 P cf9b3ba5ac8f459992fae40f75348e84: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 10:32:17.108285 5891 raft_consensus.cc:348] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b3ba5ac8f459992fae40f75348e84" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 33223 } } peers { permanent_uuid: "8b7713f10ebb487aa086154a0edd1349" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 41767 } } peers { permanent_uuid: "791e52d5e3b84ee3a8c9025dde9726ce" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 45927 } }
I20260929 10:32:17.108515 5890 ts_tablet_manager.cc:1428] T 77fa0564fd3f46fcabecbffc015440e2 P 791e52d5e3b84ee3a8c9025dde9726ce: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 10:32:17.108556 5891 raft_consensus.cc:374] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:17.108610 5891 raft_consensus.cc:729] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7713f10ebb487aa086154a0edd1349, State: Initialized, Role: FOLLOWER
I20260929 10:32:17.108904 5602 heartbeater.cc:503] Master 127.20.92.62:38591 was elected leader, sending a full tablet report...
I20260929 10:32:17.109094 5891 consensus_queue.cc:260] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b3ba5ac8f459992fae40f75348e84" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 33223 } } peers { permanent_uuid: "8b7713f10ebb487aa086154a0edd1349" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 41767 } } peers { permanent_uuid: "791e52d5e3b84ee3a8c9025dde9726ce" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 45927 } }
I20260929 10:32:17.109732 5872 heartbeater.cc:503] Master 127.20.92.62:38591 was elected leader, sending a full tablet report...
I20260929 10:32:17.110100 5891 ts_tablet_manager.cc:1428] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 10:32:17.111863 5892 raft_consensus.cc:348] T 77fa0564fd3f46fcabecbffc015440e2 P cf9b3ba5ac8f459992fae40f75348e84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b3ba5ac8f459992fae40f75348e84" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 33223 } } peers { permanent_uuid: "8b7713f10ebb487aa086154a0edd1349" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 41767 } } peers { permanent_uuid: "791e52d5e3b84ee3a8c9025dde9726ce" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 45927 } }
I20260929 10:32:17.112147 5892 raft_consensus.cc:374] T 77fa0564fd3f46fcabecbffc015440e2 P cf9b3ba5ac8f459992fae40f75348e84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:17.112227 5892 raft_consensus.cc:729] T 77fa0564fd3f46fcabecbffc015440e2 P cf9b3ba5ac8f459992fae40f75348e84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf9b3ba5ac8f459992fae40f75348e84, State: Initialized, Role: FOLLOWER
I20260929 10:32:17.112690 5892 consensus_queue.cc:260] T 77fa0564fd3f46fcabecbffc015440e2 P cf9b3ba5ac8f459992fae40f75348e84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b3ba5ac8f459992fae40f75348e84" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 33223 } } peers { permanent_uuid: "8b7713f10ebb487aa086154a0edd1349" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 41767 } } peers { permanent_uuid: "791e52d5e3b84ee3a8c9025dde9726ce" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 45927 } }
I20260929 10:32:17.113623 5737 heartbeater.cc:503] Master 127.20.92.62:38591 was elected leader, sending a full tablet report...
I20260929 10:32:17.113628 5892 ts_tablet_manager.cc:1428] T 77fa0564fd3f46fcabecbffc015440e2 P cf9b3ba5ac8f459992fae40f75348e84: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 10:32:17.119220 5897 raft_consensus.cc:482] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:32:17.119426 5897 raft_consensus.cc:504] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b3ba5ac8f459992fae40f75348e84" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 33223 } } peers { permanent_uuid: "8b7713f10ebb487aa086154a0edd1349" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 41767 } } peers { permanent_uuid: "791e52d5e3b84ee3a8c9025dde9726ce" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 45927 } }
I20260929 10:32:17.120483 5897 leader_election.cc:288] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf9b3ba5ac8f459992fae40f75348e84 (127.20.92.2:33223), 791e52d5e3b84ee3a8c9025dde9726ce (127.20.92.1:45927)
I20260929 10:32:17.124974 5874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:17.120657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:33223 (local address 127.20.92.3:40397)
0929 10:32:17.120784 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:17.120801 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:17.120887 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:17.121154 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:17.121158 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:17.121180 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:32:17.121326 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:17.121333 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.122322 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.122325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:17.123649 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:17.123660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.124509 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.124512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:17.124590 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:17.124637 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 10:32:17.124726 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 10:32:17.124798 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":45}
I20260929 10:32:17.125087 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:17.121012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:46743 (local address 127.20.92.1:45927)
0929 10:32:17.121150 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:17.121154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:17.121321 (+ 167us) server_negotiation.cc:407] Connection header received
0929 10:32:17.121443 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:17.121446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:17.121489 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:17.121569 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:32:17.121866 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.122534 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.123971 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.124563 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.124685 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:17.124756 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:17.124917 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 10:32:17.124970 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 10:32:17.125128 5899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:17.120806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:45927 (local address 127.20.92.3:46743)
0929 10:32:17.121226 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:17.121240 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:17.121342 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:17.121604 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:17.121609 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:17.121627 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:32:17.121718 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:17.121723 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.122647 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.122651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:17.123866 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:17.123873 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.124693 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.124699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:17.124794 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:17.124813 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 10:32:17.124887 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 10:32:17.124938 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":59,"thread_start_us":151,"threads_started":1}
I20260929 10:32:17.125384 5885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:17.120767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:40397 (local address 127.20.92.2:33223)
0929 10:32:17.120895 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:17.120900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:17.120938 (+ 38us) server_negotiation.cc:407] Connection header received
0929 10:32:17.120986 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:17.120989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:17.121059 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:17.121133 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:32:17.121433 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.122217 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.123772 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.124385 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.124490 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:17.124557 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:17.125160 (+ 603us) server_negotiation.cc:299] Negotiation successful
0929 10:32:17.125252 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 10:32:17.125881 5556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77fa0564fd3f46fcabecbffc015440e2" candidate_uuid: "8b7713f10ebb487aa086154a0edd1349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "791e52d5e3b84ee3a8c9025dde9726ce" is_pre_election: true
I20260929 10:32:17.126067 5691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77fa0564fd3f46fcabecbffc015440e2" candidate_uuid: "8b7713f10ebb487aa086154a0edd1349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf9b3ba5ac8f459992fae40f75348e84" is_pre_election: true
I20260929 10:32:17.126194 5556 raft_consensus.cc:2454] T 77fa0564fd3f46fcabecbffc015440e2 P 791e52d5e3b84ee3a8c9025dde9726ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b7713f10ebb487aa086154a0edd1349 in term 0.
I20260929 10:32:17.126338 5691 raft_consensus.cc:2454] T 77fa0564fd3f46fcabecbffc015440e2 P cf9b3ba5ac8f459992fae40f75348e84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b7713f10ebb487aa086154a0edd1349 in term 0.
I20260929 10:32:17.126672 5762 leader_election.cc:302] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 791e52d5e3b84ee3a8c9025dde9726ce, 8b7713f10ebb487aa086154a0edd1349; no voters:
I20260929 10:32:17.126953 5897 raft_consensus.cc:2781] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:32:17.127049 5897 raft_consensus.cc:482] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:32:17.127103 5897 raft_consensus.cc:3037] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:17.128111 5897 raft_consensus.cc:504] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b3ba5ac8f459992fae40f75348e84" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 33223 } } peers { permanent_uuid: "8b7713f10ebb487aa086154a0edd1349" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 41767 } } peers { permanent_uuid: "791e52d5e3b84ee3a8c9025dde9726ce" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 45927 } }
I20260929 10:32:17.128460 5897 leader_election.cc:288] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349 [CANDIDATE]: Term 1 election: Requested vote from peers cf9b3ba5ac8f459992fae40f75348e84 (127.20.92.2:33223), 791e52d5e3b84ee3a8c9025dde9726ce (127.20.92.1:45927)
I20260929 10:32:17.128808 5556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77fa0564fd3f46fcabecbffc015440e2" candidate_uuid: "8b7713f10ebb487aa086154a0edd1349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "791e52d5e3b84ee3a8c9025dde9726ce"
I20260929 10:32:17.128939 5556 raft_consensus.cc:3037] T 77fa0564fd3f46fcabecbffc015440e2 P 791e52d5e3b84ee3a8c9025dde9726ce [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:17.129001 5691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77fa0564fd3f46fcabecbffc015440e2" candidate_uuid: "8b7713f10ebb487aa086154a0edd1349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf9b3ba5ac8f459992fae40f75348e84"
I20260929 10:32:17.129132 5691 raft_consensus.cc:3037] T 77fa0564fd3f46fcabecbffc015440e2 P cf9b3ba5ac8f459992fae40f75348e84 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:17.130147 5556 raft_consensus.cc:2454] T 77fa0564fd3f46fcabecbffc015440e2 P 791e52d5e3b84ee3a8c9025dde9726ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b7713f10ebb487aa086154a0edd1349 in term 1.
I20260929 10:32:17.130306 5691 raft_consensus.cc:2454] T 77fa0564fd3f46fcabecbffc015440e2 P cf9b3ba5ac8f459992fae40f75348e84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b7713f10ebb487aa086154a0edd1349 in term 1.
I20260929 10:32:17.130488 5762 leader_election.cc:302] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 791e52d5e3b84ee3a8c9025dde9726ce, 8b7713f10ebb487aa086154a0edd1349; no voters:
I20260929 10:32:17.130663 5897 raft_consensus.cc:2781] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:32:17.130906 5897 raft_consensus.cc:686] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349 [term 1 LEADER]: Becoming Leader. State: Replica: 8b7713f10ebb487aa086154a0edd1349, State: Running, Role: LEADER
I20260929 10:32:17.131268 5897 consensus_queue.cc:237] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b3ba5ac8f459992fae40f75348e84" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 33223 } } peers { permanent_uuid: "8b7713f10ebb487aa086154a0edd1349" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 41767 } } peers { permanent_uuid: "791e52d5e3b84ee3a8c9025dde9726ce" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 45927 } }
I20260929 10:32:17.134445 5410 catalog_manager.cc:5495] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b7713f10ebb487aa086154a0edd1349 (127.20.92.3). New cstate: current_term: 1 leader_uuid: "8b7713f10ebb487aa086154a0edd1349" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b3ba5ac8f459992fae40f75348e84" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 33223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b7713f10ebb487aa086154a0edd1349" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 41767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "791e52d5e3b84ee3a8c9025dde9726ce" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 45927 } health_report { overall_health: UNKNOWN } } }
I20260929 10:32:17.148090 5889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:17.144397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38654 (local address 127.20.92.3:41767)
0929 10:32:17.144534 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:17.144541 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:17.144559 (+ 18us) server_negotiation.cc:407] Connection header received
0929 10:32:17.144789 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:17.144794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:17.144856 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:17.144960 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:32:17.145210 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.145725 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.146622 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.146838 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.146948 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:17.147261 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:32:17.147673 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:32:17.147802 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:17.147862 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 10:32:17.147920 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 10:32:17.154011 5691 raft_consensus.cc:1261] T 77fa0564fd3f46fcabecbffc015440e2 P cf9b3ba5ac8f459992fae40f75348e84 [term 1 FOLLOWER]: Refusing update from remote peer 8b7713f10ebb487aa086154a0edd1349: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:32:17.154032 5556 raft_consensus.cc:1261] T 77fa0564fd3f46fcabecbffc015440e2 P 791e52d5e3b84ee3a8c9025dde9726ce [term 1 FOLLOWER]: Refusing update from remote peer 8b7713f10ebb487aa086154a0edd1349: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:32:17.154798 5897 consensus_queue.cc:1035] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf9b3ba5ac8f459992fae40f75348e84" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 33223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:32:17.155089 5900 consensus_queue.cc:1035] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "791e52d5e3b84ee3a8c9025dde9726ce" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 45927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:32:17.164567 5905 mvcc.cc:205] Tried to move back new op lower bound from 7334616830574616576 to 7334616830492176384. Current Snapshot: MvccSnapshot[applied={T|T < 7334616830574616576}]
I20260929 10:32:17.164770 5904 mvcc.cc:205] Tried to move back new op lower bound from 7334616830574616576 to 7334616830492176384. Current Snapshot: MvccSnapshot[applied={T|T < 7334616830574616576}]
I20260929 10:32:17.174224 5410 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:32:17.174400 5410 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:32:17.177356 5410 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a31a9d437934b7ba634ec1d3dd03548: Sending DeleteTablet for 3 replicas of tablet 77fa0564fd3f46fcabecbffc015440e2
I20260929 10:32:17.178324 5806 tablet_service.cc:1514] Processing DeleteTablet for tablet 77fa0564fd3f46fcabecbffc015440e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:32:17 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:38652
I20260929 10:32:17.178380 5536 tablet_service.cc:1514] Processing DeleteTablet for tablet 77fa0564fd3f46fcabecbffc015440e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:32:17 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:51968
I20260929 10:32:17.178704 5671 tablet_service.cc:1514] Processing DeleteTablet for tablet 77fa0564fd3f46fcabecbffc015440e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:32:17 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:55416
I20260929 10:32:17.178902 5913 tablet_replica.cc:330] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349: stopping tablet replica
I20260929 10:32:17.179236 5914 tablet_replica.cc:330] T 77fa0564fd3f46fcabecbffc015440e2 P 791e52d5e3b84ee3a8c9025dde9726ce: stopping tablet replica
I20260929 10:32:17.179255 5913 raft_consensus.cc:2229] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:32:17.179507 5914 raft_consensus.cc:2229] T 77fa0564fd3f46fcabecbffc015440e2 P 791e52d5e3b84ee3a8c9025dde9726ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:32:17.179587 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 5472
I20260929 10:32:17.179705 5913 raft_consensus.cc:2258] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:32:17.179812 5914 raft_consensus.cc:2258] T 77fa0564fd3f46fcabecbffc015440e2 P 791e52d5e3b84ee3a8c9025dde9726ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:32:17.180310 5915 tablet_replica.cc:330] T 77fa0564fd3f46fcabecbffc015440e2 P cf9b3ba5ac8f459992fae40f75348e84: stopping tablet replica
I20260929 10:32:17.181061 5915 raft_consensus.cc:2229] T 77fa0564fd3f46fcabecbffc015440e2 P cf9b3ba5ac8f459992fae40f75348e84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:32:17.181344 5915 raft_consensus.cc:2258] T 77fa0564fd3f46fcabecbffc015440e2 P cf9b3ba5ac8f459992fae40f75348e84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:32:17.181411 5913 ts_tablet_manager.cc:1905] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:32:17.184556 5915 ts_tablet_manager.cc:1905] T 77fa0564fd3f46fcabecbffc015440e2 P cf9b3ba5ac8f459992fae40f75348e84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:32:17.184779 5913 ts_tablet_manager.cc:1918] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:32:17.184876 5913 log.cc:1196] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-2/wal/wals/77fa0564fd3f46fcabecbffc015440e2
I20260929 10:32:17.185204 5913 ts_tablet_manager.cc:1939] T 77fa0564fd3f46fcabecbffc015440e2 P 8b7713f10ebb487aa086154a0edd1349: Deleting consensus metadata
I20260929 10:32:17.186607 5395 catalog_manager.cc:4841] TS 8b7713f10ebb487aa086154a0edd1349 (127.20.92.3:41767): tablet 77fa0564fd3f46fcabecbffc015440e2 (table test-table [id=d3f3d6c0220e40ca928cf81c8b1ed89c]) successfully deleted
I20260929 10:32:17.187542 5915 ts_tablet_manager.cc:1918] T 77fa0564fd3f46fcabecbffc015440e2 P cf9b3ba5ac8f459992fae40f75348e84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:32:17.187665 5915 log.cc:1196] T 77fa0564fd3f46fcabecbffc015440e2 P cf9b3ba5ac8f459992fae40f75348e84: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677679287822-20848-0/minicluster-data/ts-1/wal/wals/77fa0564fd3f46fcabecbffc015440e2
I20260929 10:32:17.187947 5915 ts_tablet_manager.cc:1939] T 77fa0564fd3f46fcabecbffc015440e2 P cf9b3ba5ac8f459992fae40f75348e84: Deleting consensus metadata
I20260929 10:32:17.189077 5395 catalog_manager.cc:4841] TS cf9b3ba5ac8f459992fae40f75348e84 (127.20.92.2:33223): tablet 77fa0564fd3f46fcabecbffc015440e2 (table test-table [id=d3f3d6c0220e40ca928cf81c8b1ed89c]) successfully deleted
W20260929 10:32:17.189352 5397 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.92.1:45927 (error 108)
I20260929 10:32:17.190258 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 5607
I20260929 10:32:17.190516 5886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:17.190205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:45927 (local address 127.0.0.1:51970)
0929 10:32:17.190367 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:17.190434 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.1:45927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 10:32:17.194165 5397 catalog_manager.cc:4584] TS 791e52d5e3b84ee3a8c9025dde9726ce (127.20.92.1:45927): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 77fa0564fd3f46fcabecbffc015440e2: Network error: Client connection negotiation failed: client connection to 127.20.92.1:45927: connect: Connection refused (error 111)
I20260929 10:32:17.197265 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 5743
I20260929 10:32:17.202600 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 5378
2026-09-29T10:32:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2303 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T10:32:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:32:17Z Disabled control of system clock
I20260929 10:32:17.233507 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:38453
--webserver_interface=127.20.92.62
--webserver_port=0
--builtin_ntp_servers=127.20.92.20:34991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:38453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 10:32:17.345819 5920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:17.346098 5920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:17.346135 5920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:17.350221 5920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:32:17.350304 5920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:17.350322 5920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:17.350340 5920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:32:17.350360 5920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:32:17.355907 5920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:34991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:38453
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:38453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:17.357007 5920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:17.358351 5920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:17.366928 5928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:17.366937 5926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:17.366922 5925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:17.367091 5920 server_base.cc:1034] running on GCE node
I20260929 10:32:17.367925 5920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:17.369168 5920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:17.370353 5920 hybrid_clock.cc:648] HybridClock initialized: now 1790677937370341 us; error 29 us; skew 500 ppm
I20260929 10:32:17.372244 5920 webserver.cc:460] Webserver started at http://127.20.92.62:44235/ using document root <none> and password file <none>
I20260929 10:32:17.372829 5920 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:17.372923 5920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:17.373153 5920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:17.375217 5920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "c5421acd5f5f47eb8f97a5d276819fe1"
format_stamp: "Formatted at 2026-09-29 10:32:17 on dist-test-slave-9cj5"
server_key: "1275748b299e8a1afe4399da398294f4"
server_key_iv: "626cd53f10207b4eecc350ae44fbde12"
server_key_version: "encryptionkey@0"
I20260929 10:32:17.375787 5920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "c5421acd5f5f47eb8f97a5d276819fe1"
format_stamp: "Formatted at 2026-09-29 10:32:17 on dist-test-slave-9cj5"
server_key: "1275748b299e8a1afe4399da398294f4"
server_key_iv: "626cd53f10207b4eecc350ae44fbde12"
server_key_version: "encryptionkey@0"
I20260929 10:32:17.379527 5920 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 10:32:17.382081 5934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:17.383442 5920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 10:32:17.383606 5920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "c5421acd5f5f47eb8f97a5d276819fe1"
format_stamp: "Formatted at 2026-09-29 10:32:17 on dist-test-slave-9cj5"
server_key: "1275748b299e8a1afe4399da398294f4"
server_key_iv: "626cd53f10207b4eecc350ae44fbde12"
server_key_version: "encryptionkey@0"
I20260929 10:32:17.383736 5920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:17.405583 5920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:17.406383 5920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:17.406600 5920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:17.414232 5920 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:38453
I20260929 10:32:17.414244 5987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:38453 every 8 connection(s)
I20260929 10:32:17.415215 5920 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:32:17.418658 5988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:17.419625 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 5920
I20260929 10:32:17.419776 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:32:17.420080 20848 external_mini_cluster.cc:1428] Setting key 385f5ea103b4a030d469b3f013a8bede
I20260929 10:32:17.425153 5988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5421acd5f5f47eb8f97a5d276819fe1: Bootstrap starting.
I20260929 10:32:17.427757 5988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5421acd5f5f47eb8f97a5d276819fe1: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:17.428586 5988 log.cc:824] T 00000000000000000000000000000000 P c5421acd5f5f47eb8f97a5d276819fe1: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:17.429647 5991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:17.421585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36930 (local address 127.20.92.62:38453)
0929 10:32:17.422292 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:17.422309 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:17.422345 (+ 36us) server_negotiation.cc:407] Connection header received
0929 10:32:17.422995 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:17.423012 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:17.423400 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:17.423758 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:17.424226 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.425243 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.426227 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.426706 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.426894 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:17.427801 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:17.427826 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:17.427845 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:17.428278 (+ 433us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:17.428334 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:17.428354 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:17.428467 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:17.428624 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 10:32:17.428930 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":158,"threads_started":1}
I20260929 10:32:17.430771 5988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5421acd5f5f47eb8f97a5d276819fe1: No bootstrap required, opened a new log
I20260929 10:32:17.434880 5988 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5421acd5f5f47eb8f97a5d276819fe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5421acd5f5f47eb8f97a5d276819fe1" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38453 } }
I20260929 10:32:17.435161 5988 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5421acd5f5f47eb8f97a5d276819fe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:17.435258 5988 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5421acd5f5f47eb8f97a5d276819fe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5421acd5f5f47eb8f97a5d276819fe1, State: Initialized, Role: FOLLOWER
I20260929 10:32:17.435711 5988 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5421acd5f5f47eb8f97a5d276819fe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5421acd5f5f47eb8f97a5d276819fe1" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38453 } }
I20260929 10:32:17.435870 5988 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5421acd5f5f47eb8f97a5d276819fe1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:32:17.435966 5988 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5421acd5f5f47eb8f97a5d276819fe1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:32:17.436094 5988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5421acd5f5f47eb8f97a5d276819fe1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:17.437094 5988 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5421acd5f5f47eb8f97a5d276819fe1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5421acd5f5f47eb8f97a5d276819fe1" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38453 } }
I20260929 10:32:17.437456 5988 leader_election.cc:302] T 00000000000000000000000000000000 P c5421acd5f5f47eb8f97a5d276819fe1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5421acd5f5f47eb8f97a5d276819fe1; no voters:
I20260929 10:32:17.437855 5988 leader_election.cc:288] T 00000000000000000000000000000000 P c5421acd5f5f47eb8f97a5d276819fe1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:32:17.437949 5993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5421acd5f5f47eb8f97a5d276819fe1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:32:17.438222 5993 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5421acd5f5f47eb8f97a5d276819fe1 [term 1 LEADER]: Becoming Leader. State: Replica: c5421acd5f5f47eb8f97a5d276819fe1, State: Running, Role: LEADER
I20260929 10:32:17.438526 5993 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5421acd5f5f47eb8f97a5d276819fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5421acd5f5f47eb8f97a5d276819fe1" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38453 } }
I20260929 10:32:17.439225 5988 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5421acd5f5f47eb8f97a5d276819fe1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:32:17.440366 5995 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5421acd5f5f47eb8f97a5d276819fe1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5421acd5f5f47eb8f97a5d276819fe1. Latest consensus state: current_term: 1 leader_uuid: "c5421acd5f5f47eb8f97a5d276819fe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5421acd5f5f47eb8f97a5d276819fe1" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38453 } } }
I20260929 10:32:17.440513 5995 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5421acd5f5f47eb8f97a5d276819fe1 [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:17.440766 5994 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5421acd5f5f47eb8f97a5d276819fe1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5421acd5f5f47eb8f97a5d276819fe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5421acd5f5f47eb8f97a5d276819fe1" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 38453 } } }
I20260929 10:32:17.440850 5994 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5421acd5f5f47eb8f97a5d276819fe1 [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:17.440899 6002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:32:17.444121 6002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:32:17.450054 6002 catalog_manager.cc:1347] Generated new cluster ID: d49938e706a24422827953fe7db645f1
I20260929 10:32:17.450132 6002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:32:17.462867 6002 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:32:17.463939 6002 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:32:17.473560 6002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5421acd5f5f47eb8f97a5d276819fe1: Generated new TSK 0
I20260929 10:32:17.474361 6002 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:32:17.480659 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:0
--local_ip_for_outbound_sockets=127.20.92.1
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:38453
--builtin_ntp_servers=127.20.92.20:34991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 10:32:17.596635 6012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:17.596967 6012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:17.597009 6012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:17.601146 6012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:17.601208 6012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:17.601292 6012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:32:17.606222 6012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:34991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:38453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:17.607439 6012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:17.608907 6012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:17.619014 6020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:17.619021 6017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:17.619102 6012 server_base.cc:1034] running on GCE node
W20260929 10:32:17.619030 6018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:17.619828 6012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:17.620445 6012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:17.621662 6012 hybrid_clock.cc:648] HybridClock initialized: now 1790677937621643 us; error 30 us; skew 500 ppm
I20260929 10:32:17.623695 6012 webserver.cc:460] Webserver started at http://127.20.92.1:46353/ using document root <none> and password file <none>
I20260929 10:32:17.624307 6012 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:17.624402 6012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:17.624676 6012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:17.626881 6012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-0/data/instance:
uuid: "cb52c2ff89a04dd48f932c193e2f5268"
format_stamp: "Formatted at 2026-09-29 10:32:17 on dist-test-slave-9cj5"
server_key: "eb1a4c981218ea78a13f375aa4881311"
server_key_iv: "ecd20dc0c0dd0cdbc9bd07b6f9b6885c"
server_key_version: "encryptionkey@0"
I20260929 10:32:17.627485 6012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance:
uuid: "cb52c2ff89a04dd48f932c193e2f5268"
format_stamp: "Formatted at 2026-09-29 10:32:17 on dist-test-slave-9cj5"
server_key: "eb1a4c981218ea78a13f375aa4881311"
server_key_iv: "ecd20dc0c0dd0cdbc9bd07b6f9b6885c"
server_key_version: "encryptionkey@0"
I20260929 10:32:17.631207 6012 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:32:17.633934 6026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:17.635319 6012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:32:17.635500 6012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "cb52c2ff89a04dd48f932c193e2f5268"
format_stamp: "Formatted at 2026-09-29 10:32:17 on dist-test-slave-9cj5"
server_key: "eb1a4c981218ea78a13f375aa4881311"
server_key_iv: "ecd20dc0c0dd0cdbc9bd07b6f9b6885c"
server_key_version: "encryptionkey@0"
I20260929 10:32:17.635646 6012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:17.653492 6012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:17.654325 6012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:17.654568 6012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:17.655210 6012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:32:17.656229 6012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:32:17.656306 6012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:17.656384 6012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:32:17.656431 6012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:17.665931 6012 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:46467
I20260929 10:32:17.665977 6140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:46467 every 8 connection(s)
I20260929 10:32:17.666877 6012 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
I20260929 10:32:17.668237 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 6012
I20260929 10:32:17.668368 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance
I20260929 10:32:17.668689 20848 external_mini_cluster.cc:1428] Setting key c13066b23832c0528b151d708ea2393b
I20260929 10:32:17.670650 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:0
--local_ip_for_outbound_sockets=127.20.92.2
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:38453
--builtin_ntp_servers=127.20.92.20:34991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 10:32:17.676858 5991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:17.669155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:51613 (local address 127.20.92.62:38453)
0929 10:32:17.669309 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:17.669313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:17.670220 (+ 907us) server_negotiation.cc:407] Connection header received
0929 10:32:17.671749 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:17.671755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:17.671821 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:17.671913 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:17.673150 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.673690 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.674858 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.675104 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.675219 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:17.676339 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:17.676343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:17.676346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:17.676360 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:17.676382 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:17.676397 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:17.676472 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:17.676650 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 10:32:17.676716 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 10:32:17.677680 6143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:17.669292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38453 (local address 127.20.92.1:51613)
0929 10:32:17.670073 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:17.670126 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:17.671500 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:17.672119 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:17.672131 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:17.672247 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:17.672977 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:17.672993 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.673859 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.673862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:17.674729 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:17.674738 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.675262 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.675266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:17.675433 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:17.676069 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:32:17.676071 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:17.676112 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:32:17.676173 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:17.676486 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:17.676495 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:17.676501 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 10:32:17.676650 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 10:32:17.676919 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":159,"threads_started":1}
I20260929 10:32:17.678962 6141 heartbeater.cc:347] Connected to a master server at 127.20.92.62:38453
I20260929 10:32:17.679229 6141 heartbeater.cc:464] Registering TS with master...
I20260929 10:32:17.679737 6141 heartbeater.cc:511] Master 127.20.92.62:38453 requested a full tablet report, sending...
I20260929 10:32:17.681401 5951 ts_manager.cc:194] Registered new tserver with Master: cb52c2ff89a04dd48f932c193e2f5268 (127.20.92.1:46467)
I20260929 10:32:17.682996 5951 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.92.1:51613
W20260929 10:32:17.787854 6144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:17.788122 6144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:17.788161 6144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:17.792229 6144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:17.792284 6144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:17.792340 6144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:32:17.797147 6144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:34991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:38453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:17.798195 6144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:17.799475 6144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:17.810945 6150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:17.810966 6149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:17.810956 6152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:17.811010 6144 server_base.cc:1034] running on GCE node
I20260929 10:32:17.811578 6144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:17.812300 6144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:17.813494 6144 hybrid_clock.cc:648] HybridClock initialized: now 1790677937813462 us; error 51 us; skew 500 ppm
I20260929 10:32:17.815551 6144 webserver.cc:460] Webserver started at http://127.20.92.2:37817/ using document root <none> and password file <none>
I20260929 10:32:17.816146 6144 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:17.816241 6144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:17.816473 6144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:17.818626 6144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-1/data/instance:
uuid: "6c0cf5b32c69486abcd421893fcdc861"
format_stamp: "Formatted at 2026-09-29 10:32:17 on dist-test-slave-9cj5"
server_key: "7d329d8d4a52dc48aa3c473c4fed5c85"
server_key_iv: "2ca02ad8c7664fb3288205106f3fa21b"
server_key_version: "encryptionkey@0"
I20260929 10:32:17.819195 6144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance:
uuid: "6c0cf5b32c69486abcd421893fcdc861"
format_stamp: "Formatted at 2026-09-29 10:32:17 on dist-test-slave-9cj5"
server_key: "7d329d8d4a52dc48aa3c473c4fed5c85"
server_key_iv: "2ca02ad8c7664fb3288205106f3fa21b"
server_key_version: "encryptionkey@0"
I20260929 10:32:17.822880 6144 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260929 10:32:17.825302 6158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:17.826581 6144 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 10:32:17.826740 6144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "6c0cf5b32c69486abcd421893fcdc861"
format_stamp: "Formatted at 2026-09-29 10:32:17 on dist-test-slave-9cj5"
server_key: "7d329d8d4a52dc48aa3c473c4fed5c85"
server_key_iv: "2ca02ad8c7664fb3288205106f3fa21b"
server_key_version: "encryptionkey@0"
I20260929 10:32:17.826854 6144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:17.839423 6144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:17.840101 6144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:17.840312 6144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:17.840960 6144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:32:17.842024 6144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:32:17.842095 6144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:17.842192 6144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:32:17.842235 6144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:17.851311 6144 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:39579
I20260929 10:32:17.851333 6272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:39579 every 8 connection(s)
I20260929 10:32:17.852236 6144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
I20260929 10:32:17.857502 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 6144
I20260929 10:32:17.857626 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance
I20260929 10:32:17.857951 20848 external_mini_cluster.cc:1428] Setting key 5718b7a76078f66280166d1665c776af
I20260929 10:32:17.860072 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:0
--local_ip_for_outbound_sockets=127.20.92.3
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:38453
--builtin_ntp_servers=127.20.92.20:34991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 10:32:17.862941 5991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:17.854152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:50585 (local address 127.20.92.62:38453)
0929 10:32:17.854315 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:17.854318 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:17.855371 (+ 1053us) server_negotiation.cc:407] Connection header received
0929 10:32:17.856793 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:17.856796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:17.856846 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:17.856938 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:17.858159 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.858725 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.860353 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.860775 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.860897 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:17.862278 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:17.862285 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:17.862291 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:17.862311 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:17.862338 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:17.862352 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:17.862444 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:17.862657 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 10:32:17.862730 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 10:32:17.863757 6275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:17.854542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38453 (local address 127.20.92.2:50585)
0929 10:32:17.855219 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:17.855261 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:17.856568 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:17.857086 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:17.857092 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:17.857209 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:17.857973 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:17.857995 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.858857 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.858860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:17.860134 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:17.860147 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:17.860971 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:17.860975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:17.861160 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:17.861972 (+ 812us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:32:17.861974 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:17.862026 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:32:17.862103 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:17.862477 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:17.862486 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:17.862491 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 10:32:17.862629 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 10:32:17.862866 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":115,"threads_started":1}
I20260929 10:32:17.865140 6273 heartbeater.cc:347] Connected to a master server at 127.20.92.62:38453
I20260929 10:32:17.865401 6273 heartbeater.cc:464] Registering TS with master...
I20260929 10:32:17.866035 6273 heartbeater.cc:511] Master 127.20.92.62:38453 requested a full tablet report, sending...
I20260929 10:32:17.867129 5951 ts_manager.cc:194] Registered new tserver with Master: 6c0cf5b32c69486abcd421893fcdc861 (127.20.92.2:39579)
I20260929 10:32:17.867719 5951 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.92.2:50585
W20260929 10:32:17.973291 6276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:17.973620 6276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:17.973719 6276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:17.978015 6276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:17.978091 6276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:17.978173 6276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:32:17.983000 6276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:34991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:38453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:17.984091 6276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:17.985455 6276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:17.995386 6281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:17.995374 6284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:17.995407 6276 server_base.cc:1034] running on GCE node
W20260929 10:32:17.995383 6282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:17.996172 6276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:17.996881 6276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:17.998081 6276 hybrid_clock.cc:648] HybridClock initialized: now 1790677937998062 us; error 32 us; skew 500 ppm
I20260929 10:32:17.999969 6276 webserver.cc:460] Webserver started at http://127.20.92.3:38243/ using document root <none> and password file <none>
I20260929 10:32:18.000550 6276 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:18.000651 6276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:18.000882 6276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:18.002928 6276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-2/data/instance:
uuid: "9fe907ab46fc48fe9b4c686eab16b510"
format_stamp: "Formatted at 2026-09-29 10:32:17 on dist-test-slave-9cj5"
server_key: "d872b9220d9316326e928e93e582763c"
server_key_iv: "16750f5860dc7f619345a7be510727ea"
server_key_version: "encryptionkey@0"
I20260929 10:32:18.003477 6276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance:
uuid: "9fe907ab46fc48fe9b4c686eab16b510"
format_stamp: "Formatted at 2026-09-29 10:32:17 on dist-test-slave-9cj5"
server_key: "d872b9220d9316326e928e93e582763c"
server_key_iv: "16750f5860dc7f619345a7be510727ea"
server_key_version: "encryptionkey@0"
I20260929 10:32:18.007238 6276 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:32:18.009665 6290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:18.010888 6276 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:32:18.011026 6276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "9fe907ab46fc48fe9b4c686eab16b510"
format_stamp: "Formatted at 2026-09-29 10:32:17 on dist-test-slave-9cj5"
server_key: "d872b9220d9316326e928e93e582763c"
server_key_iv: "16750f5860dc7f619345a7be510727ea"
server_key_version: "encryptionkey@0"
I20260929 10:32:18.011144 6276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:18.032784 6276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:18.033489 6276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:18.033715 6276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:18.034365 6276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:32:18.035327 6276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:32:18.035400 6276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:18.035472 6276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:32:18.035524 6276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:18.044364 6276 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:38505
I20260929 10:32:18.044431 6404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:38505 every 8 connection(s)
I20260929 10:32:18.045293 6276 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:32:18.047029 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 6276
I20260929 10:32:18.047155 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance
I20260929 10:32:18.047477 20848 external_mini_cluster.cc:1428] Setting key f258930827b93c1844b8a4b9cfa85c16
I20260929 10:32:18.054476 5991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:18.047139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:35615 (local address 127.20.92.62:38453)
0929 10:32:18.047297 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:18.047302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:18.048388 (+ 1086us) server_negotiation.cc:407] Connection header received
0929 10:32:18.049927 (+ 1539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:18.049930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:18.049985 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:18.050061 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:18.050927 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:18.051454 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:18.052598 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:18.052830 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:18.052937 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:18.053972 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:18.053976 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:18.053979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:18.053992 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:18.054017 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:18.054025 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:18.054119 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:18.054277 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 10:32:18.054336 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 10:32:18.055335 6407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:18.047509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:38453 (local address 127.20.92.3:35615)
0929 10:32:18.048240 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:18.048291 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:18.049702 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:18.050229 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:18.050236 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:18.050317 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:18.050781 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:18.050792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:18.051581 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:18.051585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:18.052459 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:18.052469 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:18.052985 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:18.052990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:18.053163 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:18.053704 (+ 541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:32:18.053706 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:18.053762 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:32:18.053823 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:18.054121 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:18.054133 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:18.054141 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 10:32:18.054276 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 10:32:18.054544 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":135,"threads_started":1}
I20260929 10:32:18.056406 6405 heartbeater.cc:347] Connected to a master server at 127.20.92.62:38453
I20260929 10:32:18.056650 6405 heartbeater.cc:464] Registering TS with master...
I20260929 10:32:18.057156 6405 heartbeater.cc:511] Master 127.20.92.62:38453 requested a full tablet report, sending...
I20260929 10:32:18.058162 5951 ts_manager.cc:194] Registered new tserver with Master: 9fe907ab46fc48fe9b4c686eab16b510 (127.20.92.3:38505)
I20260929 10:32:18.058743 5951 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.92.3:35615
I20260929 10:32:18.061388 20848 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:32:18.067755 5991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:18.063431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36932 (local address 127.20.92.62:38453)
0929 10:32:18.063666 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:18.063671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:18.063833 (+ 162us) server_negotiation.cc:407] Connection header received
0929 10:32:18.064019 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:18.064025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:18.064078 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:18.064172 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:18.064487 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:18.065041 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:18.066008 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:18.066260 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:18.066383 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:18.067244 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:18.067247 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:18.067249 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:18.067261 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:18.067279 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:18.067287 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:18.067368 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:18.067520 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 10:32:18.067613 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260929 10:32:18.070881 5951 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:32:18.073846 5951 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:32:18.088279 6416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:18.082333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:39579 (local address 127.0.0.1:54596)
0929 10:32:18.082723 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:18.082777 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:18.082933 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:18.083971 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:18.083980 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:18.084007 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:32:18.084149 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:18.084164 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:18.085370 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:18.085375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:18.086826 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:18.086836 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:18.087754 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:18.087757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:18.087857 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:18.087895 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 10:32:18.088004 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 10:32:18.088117 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":88,"thread_start_us":108,"threads_started":1}
I20260929 10:32:18.088282 6417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:18.082518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54596 (local address 127.20.92.2:39579)
0929 10:32:18.082890 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:18.082902 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:18.082927 (+ 25us) server_negotiation.cc:407] Connection header received
0929 10:32:18.083555 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:18.083561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:18.083770 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:18.083922 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:32:18.084277 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:18.085258 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:18.086947 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:18.087627 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:18.087753 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:18.087974 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:18.088029 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 10:32:18.088121 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":138,"threads_started":1}
I20260929 10:32:18.088896 6420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:18.083417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:38505 (local address 127.0.0.1:52346)
0929 10:32:18.083748 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:18.083762 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:18.083843 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:18.084687 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:18.084690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:18.084708 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:32:18.084784 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:18.084789 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:18.086099 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:18.086102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:18.087100 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:18.087111 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:18.088512 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:18.088515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:18.088598 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:18.088619 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 10:32:18.088678 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 10:32:18.088738 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":69,"thread_start_us":82,"threads_started":1}
I20260929 10:32:18.089426 6421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:18.083828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52346 (local address 127.20.92.3:38505)
0929 10:32:18.084228 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:18.084241 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:18.084267 (+ 26us) server_negotiation.cc:407] Connection header received
0929 10:32:18.084337 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:18.084341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:18.084539 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:18.084687 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:32:18.084897 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:18.085982 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:18.087260 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:18.088079 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:18.088179 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:18.089120 (+ 941us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:18.089186 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 10:32:18.089280 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":158,"threads_started":1}
I20260929 10:32:18.089198 6418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:18.083110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:46467 (local address 127.0.0.1:39762)
0929 10:32:18.083344 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:18.083360 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:18.083438 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:18.084319 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:18.084323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:18.084344 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:32:18.084429 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:18.084434 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:18.085828 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:18.085831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:18.087038 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:18.087044 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:18.088109 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:18.088116 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:18.088251 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:18.088274 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 10:32:18.088327 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 10:32:18.088869 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":28,"thread_start_us":99,"threads_started":1}
I20260929 10:32:18.090054 6419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:18.083247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39762 (local address 127.20.92.1:46467)
0929 10:32:18.083916 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:18.083927 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:18.083947 (+ 20us) server_negotiation.cc:407] Connection header received
0929 10:32:18.084015 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:18.084019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:18.084192 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:18.084333 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:32:18.084541 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:18.085675 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:18.087189 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:18.087969 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:18.089271 (+ 1302us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:18.089445 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:18.089805 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 10:32:18.089907 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":115,"threads_started":1}
I20260929 10:32:18.091436 6207 tablet_service.cc:1467] Processing CreateTablet for tablet 3b8b92dc99ae4e34be4aa329b04044cf (DEFAULT_TABLE table=test-table [id=c8379c7edec64d26a203c8f4eeb058f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:32:18.092411 6207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b8b92dc99ae4e34be4aa329b04044cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:18.092114 6339 tablet_service.cc:1467] Processing CreateTablet for tablet 3b8b92dc99ae4e34be4aa329b04044cf (DEFAULT_TABLE table=test-table [id=c8379c7edec64d26a203c8f4eeb058f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:32:18.092407 6070 tablet_service.cc:1467] Processing CreateTablet for tablet 3b8b92dc99ae4e34be4aa329b04044cf (DEFAULT_TABLE table=test-table [id=c8379c7edec64d26a203c8f4eeb058f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:32:18.093091 6339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b8b92dc99ae4e34be4aa329b04044cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:18.093230 6070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b8b92dc99ae4e34be4aa329b04044cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:18.098790 6422 tablet_bootstrap.cc:492] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861: Bootstrap starting.
I20260929 10:32:18.100359 6423 tablet_bootstrap.cc:492] T 3b8b92dc99ae4e34be4aa329b04044cf P cb52c2ff89a04dd48f932c193e2f5268: Bootstrap starting.
I20260929 10:32:18.100531 6424 tablet_bootstrap.cc:492] T 3b8b92dc99ae4e34be4aa329b04044cf P 9fe907ab46fc48fe9b4c686eab16b510: Bootstrap starting.
I20260929 10:32:18.101122 6422 tablet_bootstrap.cc:654] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:18.101953 6422 log.cc:824] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:18.102483 6423 tablet_bootstrap.cc:654] T 3b8b92dc99ae4e34be4aa329b04044cf P cb52c2ff89a04dd48f932c193e2f5268: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:18.102483 6424 tablet_bootstrap.cc:654] T 3b8b92dc99ae4e34be4aa329b04044cf P 9fe907ab46fc48fe9b4c686eab16b510: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:18.103286 6424 log.cc:824] T 3b8b92dc99ae4e34be4aa329b04044cf P 9fe907ab46fc48fe9b4c686eab16b510: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:18.103349 6423 log.cc:824] T 3b8b92dc99ae4e34be4aa329b04044cf P cb52c2ff89a04dd48f932c193e2f5268: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:18.104079 6422 tablet_bootstrap.cc:492] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861: No bootstrap required, opened a new log
I20260929 10:32:18.104270 6422 ts_tablet_manager.cc:1397] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 10:32:18.105201 6423 tablet_bootstrap.cc:492] T 3b8b92dc99ae4e34be4aa329b04044cf P cb52c2ff89a04dd48f932c193e2f5268: No bootstrap required, opened a new log
I20260929 10:32:18.105238 6424 tablet_bootstrap.cc:492] T 3b8b92dc99ae4e34be4aa329b04044cf P 9fe907ab46fc48fe9b4c686eab16b510: No bootstrap required, opened a new log
I20260929 10:32:18.105396 6423 ts_tablet_manager.cc:1397] T 3b8b92dc99ae4e34be4aa329b04044cf P cb52c2ff89a04dd48f932c193e2f5268: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 10:32:18.105460 6424 ts_tablet_manager.cc:1397] T 3b8b92dc99ae4e34be4aa329b04044cf P 9fe907ab46fc48fe9b4c686eab16b510: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 10:32:18.108551 6424 raft_consensus.cc:348] T 3b8b92dc99ae4e34be4aa329b04044cf P 9fe907ab46fc48fe9b4c686eab16b510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0cf5b32c69486abcd421893fcdc861" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39579 } } peers { permanent_uuid: "cb52c2ff89a04dd48f932c193e2f5268" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46467 } } peers { permanent_uuid: "9fe907ab46fc48fe9b4c686eab16b510" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38505 } }
I20260929 10:32:18.108798 6424 raft_consensus.cc:374] T 3b8b92dc99ae4e34be4aa329b04044cf P 9fe907ab46fc48fe9b4c686eab16b510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:18.108629 6422 raft_consensus.cc:348] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0cf5b32c69486abcd421893fcdc861" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39579 } } peers { permanent_uuid: "cb52c2ff89a04dd48f932c193e2f5268" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46467 } } peers { permanent_uuid: "9fe907ab46fc48fe9b4c686eab16b510" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38505 } }
I20260929 10:32:18.108856 6424 raft_consensus.cc:729] T 3b8b92dc99ae4e34be4aa329b04044cf P 9fe907ab46fc48fe9b4c686eab16b510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fe907ab46fc48fe9b4c686eab16b510, State: Initialized, Role: FOLLOWER
I20260929 10:32:18.108867 6422 raft_consensus.cc:374] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:18.108913 6422 raft_consensus.cc:729] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c0cf5b32c69486abcd421893fcdc861, State: Initialized, Role: FOLLOWER
I20260929 10:32:18.109295 6424 consensus_queue.cc:260] T 3b8b92dc99ae4e34be4aa329b04044cf P 9fe907ab46fc48fe9b4c686eab16b510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0cf5b32c69486abcd421893fcdc861" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39579 } } peers { permanent_uuid: "cb52c2ff89a04dd48f932c193e2f5268" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46467 } } peers { permanent_uuid: "9fe907ab46fc48fe9b4c686eab16b510" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38505 } }
I20260929 10:32:18.109321 6422 consensus_queue.cc:260] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0cf5b32c69486abcd421893fcdc861" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39579 } } peers { permanent_uuid: "cb52c2ff89a04dd48f932c193e2f5268" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46467 } } peers { permanent_uuid: "9fe907ab46fc48fe9b4c686eab16b510" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38505 } }
I20260929 10:32:18.109563 6423 raft_consensus.cc:348] T 3b8b92dc99ae4e34be4aa329b04044cf P cb52c2ff89a04dd48f932c193e2f5268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0cf5b32c69486abcd421893fcdc861" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39579 } } peers { permanent_uuid: "cb52c2ff89a04dd48f932c193e2f5268" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46467 } } peers { permanent_uuid: "9fe907ab46fc48fe9b4c686eab16b510" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38505 } }
I20260929 10:32:18.109828 6423 raft_consensus.cc:374] T 3b8b92dc99ae4e34be4aa329b04044cf P cb52c2ff89a04dd48f932c193e2f5268 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:18.109889 6423 raft_consensus.cc:729] T 3b8b92dc99ae4e34be4aa329b04044cf P cb52c2ff89a04dd48f932c193e2f5268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb52c2ff89a04dd48f932c193e2f5268, State: Initialized, Role: FOLLOWER
I20260929 10:32:18.110131 6405 heartbeater.cc:503] Master 127.20.92.62:38453 was elected leader, sending a full tablet report...
I20260929 10:32:18.110358 6422 ts_tablet_manager.cc:1428] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260929 10:32:18.110321 6423 consensus_queue.cc:260] T 3b8b92dc99ae4e34be4aa329b04044cf P cb52c2ff89a04dd48f932c193e2f5268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0cf5b32c69486abcd421893fcdc861" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39579 } } peers { permanent_uuid: "cb52c2ff89a04dd48f932c193e2f5268" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46467 } } peers { permanent_uuid: "9fe907ab46fc48fe9b4c686eab16b510" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38505 } }
I20260929 10:32:18.110800 6273 heartbeater.cc:503] Master 127.20.92.62:38453 was elected leader, sending a full tablet report...
I20260929 10:32:18.111188 6423 ts_tablet_manager.cc:1428] T 3b8b92dc99ae4e34be4aa329b04044cf P cb52c2ff89a04dd48f932c193e2f5268: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 10:32:18.111188 6424 ts_tablet_manager.cc:1428] T 3b8b92dc99ae4e34be4aa329b04044cf P 9fe907ab46fc48fe9b4c686eab16b510: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260929 10:32:18.111485 6141 heartbeater.cc:503] Master 127.20.92.62:38453 was elected leader, sending a full tablet report...
W20260929 10:32:18.168195 6142 tablet.cc:2369] T 3b8b92dc99ae4e34be4aa329b04044cf P cb52c2ff89a04dd48f932c193e2f5268: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:32:18.296444 6406 tablet.cc:2369] T 3b8b92dc99ae4e34be4aa329b04044cf P 9fe907ab46fc48fe9b4c686eab16b510: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:32:18.353417 6274 tablet.cc:2369] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:32:18.370847 6429 raft_consensus.cc:482] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:32:18.371127 6429 raft_consensus.cc:504] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0cf5b32c69486abcd421893fcdc861" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39579 } } peers { permanent_uuid: "cb52c2ff89a04dd48f932c193e2f5268" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46467 } } peers { permanent_uuid: "9fe907ab46fc48fe9b4c686eab16b510" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38505 } }
I20260929 10:32:18.372437 6429 leader_election.cc:288] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb52c2ff89a04dd48f932c193e2f5268 (127.20.92.1:46467), 9fe907ab46fc48fe9b4c686eab16b510 (127.20.92.3:38505)
I20260929 10:32:18.376322 6419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:18.372801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:56753 (local address 127.20.92.1:46467)
0929 10:32:18.372931 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:18.372935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:18.373052 (+ 117us) server_negotiation.cc:407] Connection header received
0929 10:32:18.373245 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:18.373249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:18.373312 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:18.373353 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:32:18.373707 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:18.374263 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:18.375329 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:18.375754 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:18.375883 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:18.375946 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:18.376122 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 10:32:18.376195 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 10:32:18.376341 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:18.372693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:46467 (local address 127.20.92.2:56753)
0929 10:32:18.372954 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:18.372971 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:18.373077 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:18.373476 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:18.373478 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:18.373494 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:32:18.373584 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:18.373600 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:18.374385 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:18.374389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:18.375198 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:18.375208 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:18.375888 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:18.375892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:18.375998 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:18.376016 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 10:32:18.376114 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 10:32:18.376189 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":1,"thread_start_us":106,"threads_started":1}
I20260929 10:32:18.376549 6421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:18.373215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:57105 (local address 127.20.92.3:38505)
0929 10:32:18.373334 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:18.373340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:18.373396 (+ 56us) server_negotiation.cc:407] Connection header received
0929 10:32:18.373605 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:18.373608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:18.373647 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:18.373736 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:32:18.374048 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:18.374550 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:18.375582 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:18.376005 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:18.376120 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:18.376178 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:18.376370 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 10:32:18.376434 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 10:32:18.376567 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:18.373072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:38505 (local address 127.20.92.2:57105)
0929 10:32:18.373326 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:18.373340 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:18.373440 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:18.373780 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:18.373782 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:18.373799 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:32:18.373926 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:18.373932 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:18.374665 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:18.374670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:18.375458 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:18.375467 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:18.376146 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:18.376150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:18.376248 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:18.376272 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 10:32:18.376355 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 10:32:18.376423 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":81,"threads_started":1}
I20260929 10:32:18.377065 6095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8b92dc99ae4e34be4aa329b04044cf" candidate_uuid: "6c0cf5b32c69486abcd421893fcdc861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb52c2ff89a04dd48f932c193e2f5268" is_pre_election: true
I20260929 10:32:18.377144 6359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8b92dc99ae4e34be4aa329b04044cf" candidate_uuid: "6c0cf5b32c69486abcd421893fcdc861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe907ab46fc48fe9b4c686eab16b510" is_pre_election: true
I20260929 10:32:18.377403 6095 raft_consensus.cc:2454] T 3b8b92dc99ae4e34be4aa329b04044cf P cb52c2ff89a04dd48f932c193e2f5268 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c0cf5b32c69486abcd421893fcdc861 in term 0.
I20260929 10:32:18.377454 6359 raft_consensus.cc:2454] T 3b8b92dc99ae4e34be4aa329b04044cf P 9fe907ab46fc48fe9b4c686eab16b510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c0cf5b32c69486abcd421893fcdc861 in term 0.
I20260929 10:32:18.377991 6161 leader_election.cc:302] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c0cf5b32c69486abcd421893fcdc861, 9fe907ab46fc48fe9b4c686eab16b510; no voters:
I20260929 10:32:18.378260 6429 raft_consensus.cc:2781] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:32:18.378341 6429 raft_consensus.cc:482] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:32:18.378372 6429 raft_consensus.cc:3037] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:18.379460 6429 raft_consensus.cc:504] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0cf5b32c69486abcd421893fcdc861" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39579 } } peers { permanent_uuid: "cb52c2ff89a04dd48f932c193e2f5268" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46467 } } peers { permanent_uuid: "9fe907ab46fc48fe9b4c686eab16b510" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38505 } }
I20260929 10:32:18.379892 6429 leader_election.cc:288] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861 [CANDIDATE]: Term 1 election: Requested vote from peers cb52c2ff89a04dd48f932c193e2f5268 (127.20.92.1:46467), 9fe907ab46fc48fe9b4c686eab16b510 (127.20.92.3:38505)
I20260929 10:32:18.380350 6095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8b92dc99ae4e34be4aa329b04044cf" candidate_uuid: "6c0cf5b32c69486abcd421893fcdc861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb52c2ff89a04dd48f932c193e2f5268"
I20260929 10:32:18.380429 6359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8b92dc99ae4e34be4aa329b04044cf" candidate_uuid: "6c0cf5b32c69486abcd421893fcdc861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe907ab46fc48fe9b4c686eab16b510"
I20260929 10:32:18.380599 6095 raft_consensus.cc:3037] T 3b8b92dc99ae4e34be4aa329b04044cf P cb52c2ff89a04dd48f932c193e2f5268 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:18.380611 6359 raft_consensus.cc:3037] T 3b8b92dc99ae4e34be4aa329b04044cf P 9fe907ab46fc48fe9b4c686eab16b510 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:18.381651 6095 raft_consensus.cc:2454] T 3b8b92dc99ae4e34be4aa329b04044cf P cb52c2ff89a04dd48f932c193e2f5268 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c0cf5b32c69486abcd421893fcdc861 in term 1.
I20260929 10:32:18.381654 6359 raft_consensus.cc:2454] T 3b8b92dc99ae4e34be4aa329b04044cf P 9fe907ab46fc48fe9b4c686eab16b510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c0cf5b32c69486abcd421893fcdc861 in term 1.
I20260929 10:32:18.382042 6161 leader_election.cc:302] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c0cf5b32c69486abcd421893fcdc861, cb52c2ff89a04dd48f932c193e2f5268; no voters:
I20260929 10:32:18.382227 6429 raft_consensus.cc:2781] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:32:18.382478 6429 raft_consensus.cc:686] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861 [term 1 LEADER]: Becoming Leader. State: Replica: 6c0cf5b32c69486abcd421893fcdc861, State: Running, Role: LEADER
I20260929 10:32:18.382827 6429 consensus_queue.cc:237] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0cf5b32c69486abcd421893fcdc861" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39579 } } peers { permanent_uuid: "cb52c2ff89a04dd48f932c193e2f5268" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46467 } } peers { permanent_uuid: "9fe907ab46fc48fe9b4c686eab16b510" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38505 } }
I20260929 10:32:18.386257 5950 catalog_manager.cc:5495] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c0cf5b32c69486abcd421893fcdc861 (127.20.92.2). New cstate: current_term: 1 leader_uuid: "6c0cf5b32c69486abcd421893fcdc861" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0cf5b32c69486abcd421893fcdc861" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 39579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb52c2ff89a04dd48f932c193e2f5268" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fe907ab46fc48fe9b4c686eab16b510" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38505 } health_report { overall_health: UNKNOWN } } }
I20260929 10:32:18.451669 6417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:18.448028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54612 (local address 127.20.92.2:39579)
0929 10:32:18.448179 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:18.448187 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:18.448203 (+ 16us) server_negotiation.cc:407] Connection header received
0929 10:32:18.448294 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:18.448297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:18.448352 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:18.448468 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:32:18.448719 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:18.449226 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:18.450157 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:18.450377 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:18.450497 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:18.450797 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:32:18.451217 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:32:18.451354 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:18.451416 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 10:32:18.451472 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 10:32:18.457777 6359 raft_consensus.cc:1261] T 3b8b92dc99ae4e34be4aa329b04044cf P 9fe907ab46fc48fe9b4c686eab16b510 [term 1 FOLLOWER]: Refusing update from remote peer 6c0cf5b32c69486abcd421893fcdc861: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:32:18.457800 6095 raft_consensus.cc:1261] T 3b8b92dc99ae4e34be4aa329b04044cf P cb52c2ff89a04dd48f932c193e2f5268 [term 1 FOLLOWER]: Refusing update from remote peer 6c0cf5b32c69486abcd421893fcdc861: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:32:18.458647 6433 consensus_queue.cc:1035] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb52c2ff89a04dd48f932c193e2f5268" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:32:18.458866 6429 consensus_queue.cc:1035] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fe907ab46fc48fe9b4c686eab16b510" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:32:18.465315 6436 mvcc.cc:205] Tried to move back new op lower bound from 7334616835914846208 to 7334616835619057664. Current Snapshot: MvccSnapshot[applied={T|T < 7334616835914846208}]
I20260929 10:32:18.466292 6438 mvcc.cc:205] Tried to move back new op lower bound from 7334616835914846208 to 7334616835619057664. Current Snapshot: MvccSnapshot[applied={T|T < 7334616835914846208}]
I20260929 10:32:18.474386 5951 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:32:18.474579 5951 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:32:18.477188 5951 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5421acd5f5f47eb8f97a5d276819fe1: Sending DeleteTablet for 3 replicas of tablet 3b8b92dc99ae4e34be4aa329b04044cf
I20260929 10:32:18.478197 6207 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b8b92dc99ae4e34be4aa329b04044cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:32:18 UTC) from {username='slave'} at 127.0.0.1:54596
I20260929 10:32:18.478264 6070 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b8b92dc99ae4e34be4aa329b04044cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:32:18 UTC) from {username='slave'} at 127.0.0.1:39762
I20260929 10:32:18.478767 6446 tablet_replica.cc:330] T 3b8b92dc99ae4e34be4aa329b04044cf P cb52c2ff89a04dd48f932c193e2f5268: stopping tablet replica
I20260929 10:32:18.478853 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 6012
I20260929 10:32:18.478794 6339 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b8b92dc99ae4e34be4aa329b04044cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:32:18 UTC) from {username='slave'} at 127.0.0.1:52346
I20260929 10:32:18.479525 6447 tablet_replica.cc:330] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861: stopping tablet replica
I20260929 10:32:18.479853 6448 tablet_replica.cc:330] T 3b8b92dc99ae4e34be4aa329b04044cf P 9fe907ab46fc48fe9b4c686eab16b510: stopping tablet replica
I20260929 10:32:18.479863 6447 raft_consensus.cc:2229] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:32:18.480217 6448 raft_consensus.cc:2229] T 3b8b92dc99ae4e34be4aa329b04044cf P 9fe907ab46fc48fe9b4c686eab16b510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:32:18.480247 6447 raft_consensus.cc:2258] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:32:18.480620 6448 raft_consensus.cc:2258] T 3b8b92dc99ae4e34be4aa329b04044cf P 9fe907ab46fc48fe9b4c686eab16b510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:32:18.482275 6447 ts_tablet_manager.cc:1905] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:32:18.483150 6448 ts_tablet_manager.cc:1905] T 3b8b92dc99ae4e34be4aa329b04044cf P 9fe907ab46fc48fe9b4c686eab16b510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:32:18.485114 6447 ts_tablet_manager.cc:1918] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:32:18.485215 6447 log.cc:1196] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-1/wal/wals/3b8b92dc99ae4e34be4aa329b04044cf
I20260929 10:32:18.485538 6447 ts_tablet_manager.cc:1939] T 3b8b92dc99ae4e34be4aa329b04044cf P 6c0cf5b32c69486abcd421893fcdc861: Deleting consensus metadata
I20260929 10:32:18.485980 6448 ts_tablet_manager.cc:1918] T 3b8b92dc99ae4e34be4aa329b04044cf P 9fe907ab46fc48fe9b4c686eab16b510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:32:18.486085 6448 log.cc:1196] T 3b8b92dc99ae4e34be4aa329b04044cf P 9fe907ab46fc48fe9b4c686eab16b510: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677679287822-20848-0/minicluster-data/ts-2/wal/wals/3b8b92dc99ae4e34be4aa329b04044cf
I20260929 10:32:18.486382 6448 ts_tablet_manager.cc:1939] T 3b8b92dc99ae4e34be4aa329b04044cf P 9fe907ab46fc48fe9b4c686eab16b510: Deleting consensus metadata
I20260929 10:32:18.486996 5937 catalog_manager.cc:4841] TS 6c0cf5b32c69486abcd421893fcdc861 (127.20.92.2:39579): tablet 3b8b92dc99ae4e34be4aa329b04044cf (table test-table [id=c8379c7edec64d26a203c8f4eeb058f8]) successfully deleted
I20260929 10:32:18.487663 5937 catalog_manager.cc:4841] TS 9fe907ab46fc48fe9b4c686eab16b510 (127.20.92.3:38505): tablet 3b8b92dc99ae4e34be4aa329b04044cf (table test-table [id=c8379c7edec64d26a203c8f4eeb058f8]) successfully deleted
W20260929 10:32:18.487926 5937 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.92.1:46467 (error 108)
I20260929 10:32:18.488385 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 6144
I20260929 10:32:18.490716 6418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:18.490340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:46467 (local address 127.0.0.1:39778)
0929 10:32:18.490511 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:18.490609 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.1:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 10:32:18.490937 5937 catalog_manager.cc:4584] TS cb52c2ff89a04dd48f932c193e2f5268 (127.20.92.1:46467): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3b8b92dc99ae4e34be4aa329b04044cf: Network error: Client connection negotiation failed: client connection to 127.20.92.1:46467: connect: Connection refused (error 111)
I20260929 10:32:18.494562 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 6276
I20260929 10:32:18.499974 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 5920
2026-09-29T10:32:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1296 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:32:18 dist-test-slave-9cj5 krb5kdc[6452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:32:18 dist-test-slave-9cj5 krb5kdc[6452](info): set up 2 sockets
Sep 29 10:32:18 dist-test-slave-9cj5 krb5kdc[6452](info): commencing operation
krb5kdc: starting...
W20260929 10:32:19.554677 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:32:19 dist-test-slave-9cj5 krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:32:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:32:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
I20260929 10:32:19.685604 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:36003
--webserver_interface=127.20.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:36003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:32:19.802557 6466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:19.803026 6466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:19.803108 6466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:19.808555 6466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:32:19.808655 6466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:19.808687 6466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:19.808717 6466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:32:19.808754 6466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:32:19.814950 6466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:36003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:36003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:19.816109 6466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:19.817149 6466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:19.825526 6471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:19.825691 6466 server_base.cc:1034] running on GCE node
W20260929 10:32:19.825511 6472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:19.825488 6474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:19.826367 6466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:19.827487 6466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:19.828684 6466 hybrid_clock.cc:648] HybridClock initialized: now 1790677939828656 us; error 46 us; skew 500 ppm
Sep 29 10:32:19 dist-test-slave-9cj5 krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677939, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:19.831483 6466 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:32:19.832389 6466 webserver.cc:460] Webserver started at http://127.20.92.62:33437/ using document root <none> and password file <none>
I20260929 10:32:19.832932 6466 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:19.832989 6466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:19.833151 6466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:19.835199 6466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "d2e23eecbad64f3e8c1837658ce718a7"
format_stamp: "Formatted at 2026-09-29 10:32:19 on dist-test-slave-9cj5"
server_key: "9680235423ec9cd53f89570586322caa"
server_key_iv: "ab3b9ceb2d163a6714fdeebca5f47811"
server_key_version: "encryptionkey@0"
I20260929 10:32:19.835731 6466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "d2e23eecbad64f3e8c1837658ce718a7"
format_stamp: "Formatted at 2026-09-29 10:32:19 on dist-test-slave-9cj5"
server_key: "9680235423ec9cd53f89570586322caa"
server_key_iv: "ab3b9ceb2d163a6714fdeebca5f47811"
server_key_version: "encryptionkey@0"
I20260929 10:32:19.839648 6466 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 10:32:19.842291 6481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:19.843948 6466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:32:19.844072 6466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "d2e23eecbad64f3e8c1837658ce718a7"
format_stamp: "Formatted at 2026-09-29 10:32:19 on dist-test-slave-9cj5"
server_key: "9680235423ec9cd53f89570586322caa"
server_key_iv: "ab3b9ceb2d163a6714fdeebca5f47811"
server_key_version: "encryptionkey@0"
I20260929 10:32:19.844246 6466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:19.859174 6466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:19.862670 6466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:19.862886 6466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:19.870378 6466 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:36003
I20260929 10:32:19.870399 6533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:36003 every 8 connection(s)
I20260929 10:32:19.871336 6466 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:32:19.872562 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 6466
I20260929 10:32:19.872745 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:32:19.873090 20848 external_mini_cluster.cc:1428] Setting key bcaa097e09c6b6ff15a37d2fac180680
I20260929 10:32:19.874522 6534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:19.880306 6534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2e23eecbad64f3e8c1837658ce718a7: Bootstrap starting.
Sep 29 10:32:19 dist-test-slave-9cj5 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:32:19.882983 6534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2e23eecbad64f3e8c1837658ce718a7: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:19.884065 6534 log.cc:824] T 00000000000000000000000000000000 P d2e23eecbad64f3e8c1837658ce718a7: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:19.886459 6534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2e23eecbad64f3e8c1837658ce718a7: No bootstrap required, opened a new log
I20260929 10:32:19.888425 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:19.874529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33602 (local address 127.20.92.62:36003)
0929 10:32:19.875022 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:19.875042 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:19.875087 (+ 45us) server_negotiation.cc:407] Connection header received
0929 10:32:19.875759 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:19.875775 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:19.876390 (+ 615us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:19.876862 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:19.877693 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:19.878759 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:19.879658 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:19.880155 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:19.880327 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:19.883047 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:19.883052 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:19.883067 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:19.883121 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:19.885313 (+ 2192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:19.885895 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:19.885901 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:19.885905 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:19.886017 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:19.886334 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:19.886336 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:19.886337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:19.886861 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:19.887007 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:19.887342 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 10:32:19.887700 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":145,"threads_started":1}
I20260929 10:32:19.890403 6534 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2e23eecbad64f3e8c1837658ce718a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e23eecbad64f3e8c1837658ce718a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 36003 } }
I20260929 10:32:19.890678 6534 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2e23eecbad64f3e8c1837658ce718a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:19.890734 6534 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2e23eecbad64f3e8c1837658ce718a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2e23eecbad64f3e8c1837658ce718a7, State: Initialized, Role: FOLLOWER
I20260929 10:32:19.891167 6534 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2e23eecbad64f3e8c1837658ce718a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e23eecbad64f3e8c1837658ce718a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 36003 } }
I20260929 10:32:19.891322 6534 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2e23eecbad64f3e8c1837658ce718a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:32:19.891407 6534 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2e23eecbad64f3e8c1837658ce718a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:32:19.891531 6534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2e23eecbad64f3e8c1837658ce718a7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:19.892498 6534 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2e23eecbad64f3e8c1837658ce718a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e23eecbad64f3e8c1837658ce718a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 36003 } }
I20260929 10:32:19.892850 6534 leader_election.cc:302] T 00000000000000000000000000000000 P d2e23eecbad64f3e8c1837658ce718a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2e23eecbad64f3e8c1837658ce718a7; no voters:
I20260929 10:32:19.893244 6534 leader_election.cc:288] T 00000000000000000000000000000000 P d2e23eecbad64f3e8c1837658ce718a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:32:19.893308 6539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2e23eecbad64f3e8c1837658ce718a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:32:19.893529 6539 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2e23eecbad64f3e8c1837658ce718a7 [term 1 LEADER]: Becoming Leader. State: Replica: d2e23eecbad64f3e8c1837658ce718a7, State: Running, Role: LEADER
I20260929 10:32:19.893955 6539 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2e23eecbad64f3e8c1837658ce718a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e23eecbad64f3e8c1837658ce718a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 36003 } }
I20260929 10:32:19.894760 6534 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2e23eecbad64f3e8c1837658ce718a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:32:19.895736 6541 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2e23eecbad64f3e8c1837658ce718a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2e23eecbad64f3e8c1837658ce718a7. Latest consensus state: current_term: 1 leader_uuid: "d2e23eecbad64f3e8c1837658ce718a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e23eecbad64f3e8c1837658ce718a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 36003 } } }
I20260929 10:32:19.895884 6541 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2e23eecbad64f3e8c1837658ce718a7 [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:19.896308 6548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:32:19.897183 6540 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2e23eecbad64f3e8c1837658ce718a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2e23eecbad64f3e8c1837658ce718a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e23eecbad64f3e8c1837658ce718a7" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 36003 } } }
I20260929 10:32:19.897279 6540 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2e23eecbad64f3e8c1837658ce718a7 [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:19.899228 6548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:32:19.905223 6548 catalog_manager.cc:1347] Generated new cluster ID: 1ec6417714764ca996b696320a01e16e
I20260929 10:32:19.905299 6548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:32:19.920583 6548 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:32:19.922087 6548 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:32:19.936182 6548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2e23eecbad64f3e8c1837658ce718a7: Generated new TSK 0
I20260929 10:32:19.936975 6548 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
I20260929 10:32:19.977969 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:0
--local_ip_for_outbound_sockets=127.20.92.1
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:36003
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:32:20.092945 6560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:20.093267 6560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:20.093345 6560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:20.097644 6560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:20.097738 6560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:20.097851 6560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:32:20.103402 6560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:36003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:20.104508 6560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:20.105544 6560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:20.115280 6566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:20.115337 6560 server_base.cc:1034] running on GCE node
W20260929 10:32:20.115322 6565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:20.115499 6568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:20.115962 6560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:20.116659 6560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:20.117823 6560 hybrid_clock.cc:648] HybridClock initialized: now 1790677940117813 us; error 32 us; skew 500 ppm
Sep 29 10:32:20 dist-test-slave-9cj5 krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677940, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:20.124452 6560 init.cc:377] Logged in from keytab as kudu/127.20.92.1@KRBTEST.COM (short username kudu)
I20260929 10:32:20.125391 6560 webserver.cc:460] Webserver started at http://127.20.92.1:32775/ using document root <none> and password file <none>
I20260929 10:32:20.126008 6560 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:20.126080 6560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:20.126255 6560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:20.128324 6560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-0/data/instance:
uuid: "fc4d959c27ef47a1a275928825dfb868"
format_stamp: "Formatted at 2026-09-29 10:32:20 on dist-test-slave-9cj5"
server_key: "f332e8504412fa31e93f9e8a979f93b2"
server_key_iv: "791312520114b334ed0759c05c673e31"
server_key_version: "encryptionkey@0"
I20260929 10:32:20.128865 6560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance:
uuid: "fc4d959c27ef47a1a275928825dfb868"
format_stamp: "Formatted at 2026-09-29 10:32:20 on dist-test-slave-9cj5"
server_key: "f332e8504412fa31e93f9e8a979f93b2"
server_key_iv: "791312520114b334ed0759c05c673e31"
server_key_version: "encryptionkey@0"
I20260929 10:32:20.132735 6560 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 10:32:20.135202 6575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:20.136390 6560 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:32:20.136484 6560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "fc4d959c27ef47a1a275928825dfb868"
format_stamp: "Formatted at 2026-09-29 10:32:20 on dist-test-slave-9cj5"
server_key: "f332e8504412fa31e93f9e8a979f93b2"
server_key_iv: "791312520114b334ed0759c05c673e31"
server_key_version: "encryptionkey@0"
I20260929 10:32:20.136569 6560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:20.146046 6560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:20.149309 6560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:20.149506 6560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:20.150200 6560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:32:20.151049 6560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:32:20.151093 6560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:20.151160 6560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:32:20.151185 6560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:20.160027 6560 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:34523
I20260929 10:32:20.160054 6689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:34523 every 8 connection(s)
I20260929 10:32:20.160966 6560 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
I20260929 10:32:20.164835 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 6560
I20260929 10:32:20.164994 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance
I20260929 10:32:20.165446 20848 external_mini_cluster.cc:1428] Setting key d918c27a6e38d01bc315b4a0bdb5b998
Sep 29 10:32:20 dist-test-slave-9cj5 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677940, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.92.2@KRBTEST.COM; defaulting to no policy
I20260929 10:32:20.177016 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:20.162999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:46671 (local address 127.20.92.62:36003)
0929 10:32:20.163151 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:20.163155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:20.163929 (+ 774us) server_negotiation.cc:407] Connection header received
0929 10:32:20.165985 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:20.165988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:20.166052 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:20.166159 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:20.167599 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.168120 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.169467 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.169924 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.170049 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:20.172873 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:20.172877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:20.172885 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:20.172911 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:20.174714 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:20.175347 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:20.175351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:20.175354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:20.175439 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:20.175808 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:20.175813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:20.175815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:20.176028 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:20.176143 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:20.176759 (+ 616us) server_negotiation.cc:299] Negotiation successful
0929 10:32:20.176876 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.2@KRBTEST.COM" created.
I20260929 10:32:20.178136 6692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:20.163317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:36003 (local address 127.20.92.1:46671)
0929 10:32:20.163782 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:20.163824 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:20.165694 (+ 1870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:20.166336 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:20.166344 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:20.166716 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:20.167404 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:20.167426 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.168255 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.168262 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:20.169340 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:20.169349 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.170109 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.170112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:20.170262 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:20.170996 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:32:20.171025 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:20.172688 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:20.174846 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:20.174852 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:20.174870 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:20.175240 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:20.175541 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:20.175545 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:20.175546 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:20.175710 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:20.176173 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:20.176178 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:20.176477 (+ 299us) client_negotiation.cc:769] Sending connection context
0929 10:32:20.176671 (+ 194us) client_negotiation.cc:240] Negotiation successful
0929 10:32:20.176991 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":136,"threads_started":1}
I20260929 10:32:20.179597 6690 heartbeater.cc:347] Connected to a master server at 127.20.92.62:36003
I20260929 10:32:20.179878 6690 heartbeater.cc:464] Registering TS with master...
I20260929 10:32:20.180428 6690 heartbeater.cc:511] Master 127.20.92.62:36003 requested a full tablet report, sending...
I20260929 10:32:20.182307 6497 ts_manager.cc:194] Registered new tserver with Master: fc4d959c27ef47a1a275928825dfb868 (127.20.92.1:34523)
I20260929 10:32:20.183492 6497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.1@KRBTEST.COM'} at 127.20.92.1:46671
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
I20260929 10:32:20.196769 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:0
--local_ip_for_outbound_sockets=127.20.92.2
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:36003
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:32:20.312096 6695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:20.312381 6695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:20.312465 6695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:20.316498 6695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:20.316562 6695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:20.316622 6695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:32:20.321946 6695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:36003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:20.323024 6695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:20.323971 6695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:20.333665 6703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:20.333727 6695 server_base.cc:1034] running on GCE node
W20260929 10:32:20.333675 6700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:20.333673 6701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:20.334640 6695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:20.335403 6695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:20.336589 6695 hybrid_clock.cc:648] HybridClock initialized: now 1790677940336557 us; error 44 us; skew 500 ppm
Sep 29 10:32:20 dist-test-slave-9cj5 krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677940, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:20.339417 6695 init.cc:377] Logged in from keytab as kudu/127.20.92.2@KRBTEST.COM (short username kudu)
I20260929 10:32:20.340339 6695 webserver.cc:460] Webserver started at http://127.20.92.2:37805/ using document root <none> and password file <none>
I20260929 10:32:20.340929 6695 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:20.341023 6695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:20.341238 6695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:20.343443 6695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-1/data/instance:
uuid: "f7cf315c39b041fbb658859bddff28db"
format_stamp: "Formatted at 2026-09-29 10:32:20 on dist-test-slave-9cj5"
server_key: "64004374772fe6e38baf3a983cc06436"
server_key_iv: "1a07485920d19b9b8378a0d9c8db3aca"
server_key_version: "encryptionkey@0"
I20260929 10:32:20.344009 6695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance:
uuid: "f7cf315c39b041fbb658859bddff28db"
format_stamp: "Formatted at 2026-09-29 10:32:20 on dist-test-slave-9cj5"
server_key: "64004374772fe6e38baf3a983cc06436"
server_key_iv: "1a07485920d19b9b8378a0d9c8db3aca"
server_key_version: "encryptionkey@0"
I20260929 10:32:20.347862 6695 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:32:20.350350 6710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:20.351516 6695 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:32:20.351692 6695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "f7cf315c39b041fbb658859bddff28db"
format_stamp: "Formatted at 2026-09-29 10:32:20 on dist-test-slave-9cj5"
server_key: "64004374772fe6e38baf3a983cc06436"
server_key_iv: "1a07485920d19b9b8378a0d9c8db3aca"
server_key_version: "encryptionkey@0"
I20260929 10:32:20.351809 6695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:20.366357 6695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:20.369796 6695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:20.370012 6695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:20.370638 6695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:32:20.371481 6695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:32:20.371562 6695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:20.371655 6695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:32:20.371708 6695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:20.382305 6695 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:41253
I20260929 10:32:20.382323 6825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:41253 every 8 connection(s)
I20260929 10:32:20.383281 6695 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
I20260929 10:32:20.383764 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 6695
I20260929 10:32:20.383903 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance
I20260929 10:32:20.384387 20848 external_mini_cluster.cc:1428] Setting key 4e2a695e5d05ccc9a18510b216ea4e1c
WARNING: no policy specified for kudu/127.20.92.3@KRBTEST.COM; defaulting to no policy
Sep 29 10:32:20 dist-test-slave-9cj5 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677940, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.3@KRBTEST.COM" created.
I20260929 10:32:20.401355 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:20.387017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:37683 (local address 127.20.92.62:36003)
0929 10:32:20.387200 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:20.387207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:20.387225 (+ 18us) server_negotiation.cc:407] Connection header received
0929 10:32:20.388250 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:20.388256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:20.388323 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:20.388423 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:20.389913 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.390857 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.392482 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.392939 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.393104 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:20.396788 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:20.396791 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:20.396794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:20.396822 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:20.398678 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:20.399297 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:20.399301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:20.399305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:20.399389 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:20.399876 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:20.399884 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:20.399889 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:20.400206 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:20.400335 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:20.401090 (+ 755us) server_negotiation.cc:299] Negotiation successful
0929 10:32:20.401216 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 10:32:20.402649 6828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:20.385894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:36003 (local address 127.20.92.2:37683)
0929 10:32:20.386406 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:20.386449 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:20.387981 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:20.388664 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:20.388677 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:20.389188 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:20.389684 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:20.389698 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.391214 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.391219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:20.392293 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:20.392309 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.393151 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.393154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:20.393324 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:20.394143 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:32:20.394162 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:20.396555 (+ 2393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:20.398821 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:20.398829 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:20.398848 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:20.399176 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:20.399488 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:20.399492 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:20.399494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:20.399718 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:20.400354 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:20.400363 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:20.400757 (+ 394us) client_negotiation.cc:769] Sending connection context
0929 10:32:20.400986 (+ 229us) client_negotiation.cc:240] Negotiation successful
0929 10:32:20.401399 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":165,"threads_started":1}
I20260929 10:32:20.404078 6826 heartbeater.cc:347] Connected to a master server at 127.20.92.62:36003
I20260929 10:32:20.404376 6826 heartbeater.cc:464] Registering TS with master...
I20260929 10:32:20.404927 6826 heartbeater.cc:511] Master 127.20.92.62:36003 requested a full tablet report, sending...
I20260929 10:32:20.406164 6497 ts_manager.cc:194] Registered new tserver with Master: f7cf315c39b041fbb658859bddff28db (127.20.92.2:41253)
I20260929 10:32:20.406839 6497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.2@KRBTEST.COM'} at 127.20.92.2:37683
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
I20260929 10:32:20.418308 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:0
--local_ip_for_outbound_sockets=127.20.92.3
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:36003
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:32:20.531097 6831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:20.531366 6831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:20.531442 6831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:20.535584 6831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:20.535645 6831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:20.535746 6831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:32:20.541404 6831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:36003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:20.542531 6831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:20.543536 6831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:20.553278 6839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:20.553288 6837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:20.553303 6836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:20.553490 6831 server_base.cc:1034] running on GCE node
I20260929 10:32:20.554031 6831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:20.554684 6831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:20.555841 6831 hybrid_clock.cc:648] HybridClock initialized: now 1790677940555824 us; error 30 us; skew 500 ppm
Sep 29 10:32:20 dist-test-slave-9cj5 krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677940, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:20.558678 6831 init.cc:377] Logged in from keytab as kudu/127.20.92.3@KRBTEST.COM (short username kudu)
I20260929 10:32:20.559669 6831 webserver.cc:460] Webserver started at http://127.20.92.3:45517/ using document root <none> and password file <none>
I20260929 10:32:20.560282 6831 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:20.560348 6831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:20.560581 6831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:20.562896 6831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-2/data/instance:
uuid: "d3760ffaa2bf45fab30b6903b2e2ac8f"
format_stamp: "Formatted at 2026-09-29 10:32:20 on dist-test-slave-9cj5"
server_key: "ccc798900d3149c07b66102ffca7c84a"
server_key_iv: "2bfdfe2116e3baeea4aba9dc71e5d4e3"
server_key_version: "encryptionkey@0"
I20260929 10:32:20.563524 6831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance:
uuid: "d3760ffaa2bf45fab30b6903b2e2ac8f"
format_stamp: "Formatted at 2026-09-29 10:32:20 on dist-test-slave-9cj5"
server_key: "ccc798900d3149c07b66102ffca7c84a"
server_key_iv: "2bfdfe2116e3baeea4aba9dc71e5d4e3"
server_key_version: "encryptionkey@0"
I20260929 10:32:20.567759 6831 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:32:20.570518 6846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:20.571903 6831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:32:20.572072 6831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "d3760ffaa2bf45fab30b6903b2e2ac8f"
format_stamp: "Formatted at 2026-09-29 10:32:20 on dist-test-slave-9cj5"
server_key: "ccc798900d3149c07b66102ffca7c84a"
server_key_iv: "2bfdfe2116e3baeea4aba9dc71e5d4e3"
server_key_version: "encryptionkey@0"
I20260929 10:32:20.572197 6831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:20.598821 6831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:20.602105 6831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:20.602337 6831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:20.602942 6831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:32:20.603766 6831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:32:20.603842 6831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:20.603915 6831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:32:20.603958 6831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:20.612382 6831 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:45155
I20260929 10:32:20.612435 6960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:45155 every 8 connection(s)
I20260929 10:32:20.613300 6831 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:32:20.615412 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 6831
I20260929 10:32:20.615597 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance
I20260929 10:32:20.615922 20848 external_mini_cluster.cc:1428] Setting key e6edb2ba271b63ea514c3a05d68de260
Sep 29 10:32:20 dist-test-slave-9cj5 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677940, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:32:20.626982 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:20.615196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:57931 (local address 127.20.92.62:36003)
0929 10:32:20.615350 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:20.615355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:20.616351 (+ 996us) server_negotiation.cc:407] Connection header received
0929 10:32:20.617927 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:20.617930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:20.617984 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:20.618064 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:20.619180 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.619716 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.620822 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.621066 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.621179 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:20.623601 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:20.623605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:20.623607 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:20.623631 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:20.625089 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:20.625554 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:20.625556 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:20.625558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:20.625630 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:20.626019 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:20.626022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:20.626023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:20.626201 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:20.626279 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:20.626760 (+ 481us) server_negotiation.cc:299] Negotiation successful
0929 10:32:20.626861 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 10:32:20.627844 6963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:20.615617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:36003 (local address 127.20.92.3:57931)
0929 10:32:20.616180 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:20.616236 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:20.617682 (+ 1446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:20.618227 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:20.618234 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:20.618581 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:20.619043 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:20.619056 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.619841 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.619844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:20.620699 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:20.620707 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.621215 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.621219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:20.621378 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:20.621952 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:32:20.621967 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:20.623446 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:20.625191 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:20.625195 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:20.625208 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:20.625462 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:20.625732 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:20.625735 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:20.625736 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:20.625936 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:20.626289 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:20.626294 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:20.626523 (+ 229us) client_negotiation.cc:769] Sending connection context
0929 10:32:20.626682 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 10:32:20.626941 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":161,"threads_started":1}
I20260929 10:32:20.629092 6961 heartbeater.cc:347] Connected to a master server at 127.20.92.62:36003
I20260929 10:32:20.629356 6961 heartbeater.cc:464] Registering TS with master...
I20260929 10:32:20.629941 6961 heartbeater.cc:511] Master 127.20.92.62:36003 requested a full tablet report, sending...
I20260929 10:32:20.630924 6497 ts_manager.cc:194] Registered new tserver with Master: d3760ffaa2bf45fab30b6903b2e2ac8f (127.20.92.3:45155)
I20260929 10:32:20.631493 6497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.3@KRBTEST.COM'} at 127.20.92.3:57931
I20260929 10:32:20.641390 20848 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:32:20.651011 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:20.643394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33608 (local address 127.20.92.62:36003)
0929 10:32:20.643534 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:20.643539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:20.643715 (+ 176us) server_negotiation.cc:407] Connection header received
0929 10:32:20.643807 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:20.643809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:20.643851 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:20.643915 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:20.644630 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.645155 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.646048 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.646257 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.646335 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:20.647820 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:20.647822 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:20.647824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:20.647846 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:20.649394 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:20.649894 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:20.649897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:20.649898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:20.649967 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:20.650241 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:20.650244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:20.650245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:20.650425 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:20.650534 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:20.650742 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 10:32:20.650813 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 10:32:20.654022 6497 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:32:20.657193 6497 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:32:20.673441 6975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:20.666898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:41253 (local address 127.0.0.1:46812)
0929 10:32:20.668116 (+ 1218us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:20.668131 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:20.668216 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:20.669198 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:20.669202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:20.669224 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:32:20.669337 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:20.669344 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.670779 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.670786 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:20.671947 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:20.672000 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.672869 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.672873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:20.672989 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:20.673025 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 10:32:20.673141 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 10:32:20.673269 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1158,"spinlock_wait_cycles":1280,"thread_start_us":431,"threads_started":1}
I20260929 10:32:20.673483 6972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:20.666014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:34523 (local address 127.0.0.1:40048)
0929 10:32:20.666525 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:20.666573 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:20.666732 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:20.668076 (+ 1344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:20.668083 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:20.668111 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:32:20.668378 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:20.668384 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.670353 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.670357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:20.671889 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:20.671900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.673149 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.673152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:20.673241 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:20.673260 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 10:32:20.673319 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 10:32:20.673372 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":1,"spinlock_wait_cycles":2816,"thread_start_us":91,"threads_started":1}
I20260929 10:32:20.673682 6974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:20.666222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40048 (local address 127.20.92.1:34523)
0929 10:32:20.666866 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:20.666884 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:20.666909 (+ 25us) server_negotiation.cc:407] Connection header received
0929 10:32:20.666999 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:20.667005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:20.667717 (+ 712us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:20.668561 (+ 844us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:32:20.668713 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.670198 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.672048 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.673011 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.673156 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:20.673305 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:20.673366 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 10:32:20.673470 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":107,"threads_started":1}
I20260929 10:32:20.674165 6977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:20.667082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46812 (local address 127.20.92.2:41253)
0929 10:32:20.668546 (+ 1464us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:20.668557 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:20.668580 (+ 23us) server_negotiation.cc:407] Connection header received
0929 10:32:20.668683 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:20.668690 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:20.668847 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:20.668994 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:32:20.669492 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.670639 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.672129 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.672737 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.672869 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:20.673025 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:20.673948 (+ 923us) server_negotiation.cc:299] Negotiation successful
0929 10:32:20.674051 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1360,"thread_start_us":254,"threads_started":1}
I20260929 10:32:20.674778 6976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:20.667367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45170 (local address 127.20.92.3:45155)
0929 10:32:20.667932 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:20.667974 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:20.668083 (+ 109us) server_negotiation.cc:407] Connection header received
0929 10:32:20.668336 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:20.668342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:20.668610 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:20.668733 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:32:20.669535 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.670779 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.672221 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.673056 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.673158 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:20.673305 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:20.673661 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 10:32:20.674639 (+ 978us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":162,"threads_started":1}
I20260929 10:32:20.677188 6973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:20.666224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:45155 (local address 127.0.0.1:45170)
0929 10:32:20.667976 (+ 1752us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:20.667991 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:20.668181 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:20.668929 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:20.668934 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:20.668956 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:32:20.669055 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:20.669061 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.670956 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.670960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:20.672095 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:20.672105 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.673377 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.673380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:20.673473 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:20.673498 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 10:32:20.674985 (+ 1487us) client_negotiation.cc:240] Negotiation successful
0929 10:32:20.675066 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1686,"thread_start_us":112,"threads_started":1}
I20260929 10:32:20.676826 6624 tablet_service.cc:1467] Processing CreateTablet for tablet 3b0a74b87f5647c387a6d1d5281c6d82 (DEFAULT_TABLE table=test-table [id=7c9c4677466a4b048e2fa226464ea904]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:32:20.676829 6758 tablet_service.cc:1467] Processing CreateTablet for tablet 3b0a74b87f5647c387a6d1d5281c6d82 (DEFAULT_TABLE table=test-table [id=7c9c4677466a4b048e2fa226464ea904]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:32:20.677906 6624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b0a74b87f5647c387a6d1d5281c6d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:20.677942 6758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b0a74b87f5647c387a6d1d5281c6d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:20.679993 6894 tablet_service.cc:1467] Processing CreateTablet for tablet 3b0a74b87f5647c387a6d1d5281c6d82 (DEFAULT_TABLE table=test-table [id=7c9c4677466a4b048e2fa226464ea904]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:32:20.681007 6894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b0a74b87f5647c387a6d1d5281c6d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:20.684777 6978 tablet_bootstrap.cc:492] T 3b0a74b87f5647c387a6d1d5281c6d82 P f7cf315c39b041fbb658859bddff28db: Bootstrap starting.
I20260929 10:32:20.685840 6979 tablet_bootstrap.cc:492] T 3b0a74b87f5647c387a6d1d5281c6d82 P fc4d959c27ef47a1a275928825dfb868: Bootstrap starting.
I20260929 10:32:20.687567 6978 tablet_bootstrap.cc:654] T 3b0a74b87f5647c387a6d1d5281c6d82 P f7cf315c39b041fbb658859bddff28db: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:20.687932 6979 tablet_bootstrap.cc:654] T 3b0a74b87f5647c387a6d1d5281c6d82 P fc4d959c27ef47a1a275928825dfb868: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:20.688197 6980 tablet_bootstrap.cc:492] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f: Bootstrap starting.
I20260929 10:32:20.688575 6978 log.cc:824] T 3b0a74b87f5647c387a6d1d5281c6d82 P f7cf315c39b041fbb658859bddff28db: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:20.688735 6979 log.cc:824] T 3b0a74b87f5647c387a6d1d5281c6d82 P fc4d959c27ef47a1a275928825dfb868: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:20.690697 6980 tablet_bootstrap.cc:654] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:20.690737 6979 tablet_bootstrap.cc:492] T 3b0a74b87f5647c387a6d1d5281c6d82 P fc4d959c27ef47a1a275928825dfb868: No bootstrap required, opened a new log
I20260929 10:32:20.690829 6978 tablet_bootstrap.cc:492] T 3b0a74b87f5647c387a6d1d5281c6d82 P f7cf315c39b041fbb658859bddff28db: No bootstrap required, opened a new log
I20260929 10:32:20.690939 6979 ts_tablet_manager.cc:1397] T 3b0a74b87f5647c387a6d1d5281c6d82 P fc4d959c27ef47a1a275928825dfb868: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 10:32:20.691039 6978 ts_tablet_manager.cc:1397] T 3b0a74b87f5647c387a6d1d5281c6d82 P f7cf315c39b041fbb658859bddff28db: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 10:32:20.692060 6980 log.cc:824] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:20.694259 6980 tablet_bootstrap.cc:492] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f: No bootstrap required, opened a new log
I20260929 10:32:20.694461 6980 ts_tablet_manager.cc:1397] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 10:32:20.694839 6979 raft_consensus.cc:348] T 3b0a74b87f5647c387a6d1d5281c6d82 P fc4d959c27ef47a1a275928825dfb868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc4d959c27ef47a1a275928825dfb868" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 34523 } } peers { permanent_uuid: "d3760ffaa2bf45fab30b6903b2e2ac8f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 45155 } } peers { permanent_uuid: "f7cf315c39b041fbb658859bddff28db" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41253 } }
I20260929 10:32:20.695076 6979 raft_consensus.cc:374] T 3b0a74b87f5647c387a6d1d5281c6d82 P fc4d959c27ef47a1a275928825dfb868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:20.695127 6979 raft_consensus.cc:729] T 3b0a74b87f5647c387a6d1d5281c6d82 P fc4d959c27ef47a1a275928825dfb868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc4d959c27ef47a1a275928825dfb868, State: Initialized, Role: FOLLOWER
I20260929 10:32:20.695653 6979 consensus_queue.cc:260] T 3b0a74b87f5647c387a6d1d5281c6d82 P fc4d959c27ef47a1a275928825dfb868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc4d959c27ef47a1a275928825dfb868" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 34523 } } peers { permanent_uuid: "d3760ffaa2bf45fab30b6903b2e2ac8f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 45155 } } peers { permanent_uuid: "f7cf315c39b041fbb658859bddff28db" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41253 } }
I20260929 10:32:20.695516 6978 raft_consensus.cc:348] T 3b0a74b87f5647c387a6d1d5281c6d82 P f7cf315c39b041fbb658859bddff28db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc4d959c27ef47a1a275928825dfb868" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 34523 } } peers { permanent_uuid: "d3760ffaa2bf45fab30b6903b2e2ac8f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 45155 } } peers { permanent_uuid: "f7cf315c39b041fbb658859bddff28db" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41253 } }
I20260929 10:32:20.695828 6978 raft_consensus.cc:374] T 3b0a74b87f5647c387a6d1d5281c6d82 P f7cf315c39b041fbb658859bddff28db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:20.695891 6978 raft_consensus.cc:729] T 3b0a74b87f5647c387a6d1d5281c6d82 P f7cf315c39b041fbb658859bddff28db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7cf315c39b041fbb658859bddff28db, State: Initialized, Role: FOLLOWER
I20260929 10:32:20.696329 6978 consensus_queue.cc:260] T 3b0a74b87f5647c387a6d1d5281c6d82 P f7cf315c39b041fbb658859bddff28db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc4d959c27ef47a1a275928825dfb868" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 34523 } } peers { permanent_uuid: "d3760ffaa2bf45fab30b6903b2e2ac8f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 45155 } } peers { permanent_uuid: "f7cf315c39b041fbb658859bddff28db" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41253 } }
I20260929 10:32:20.696626 6979 ts_tablet_manager.cc:1428] T 3b0a74b87f5647c387a6d1d5281c6d82 P fc4d959c27ef47a1a275928825dfb868: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 10:32:20.696985 6690 heartbeater.cc:503] Master 127.20.92.62:36003 was elected leader, sending a full tablet report...
I20260929 10:32:20.697289 6978 ts_tablet_manager.cc:1428] T 3b0a74b87f5647c387a6d1d5281c6d82 P f7cf315c39b041fbb658859bddff28db: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 10:32:20.697641 6826 heartbeater.cc:503] Master 127.20.92.62:36003 was elected leader, sending a full tablet report...
I20260929 10:32:20.698935 6980 raft_consensus.cc:348] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc4d959c27ef47a1a275928825dfb868" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 34523 } } peers { permanent_uuid: "d3760ffaa2bf45fab30b6903b2e2ac8f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 45155 } } peers { permanent_uuid: "f7cf315c39b041fbb658859bddff28db" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41253 } }
I20260929 10:32:20.699221 6980 raft_consensus.cc:374] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:20.699283 6980 raft_consensus.cc:729] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3760ffaa2bf45fab30b6903b2e2ac8f, State: Initialized, Role: FOLLOWER
I20260929 10:32:20.699751 6980 consensus_queue.cc:260] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc4d959c27ef47a1a275928825dfb868" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 34523 } } peers { permanent_uuid: "d3760ffaa2bf45fab30b6903b2e2ac8f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 45155 } } peers { permanent_uuid: "f7cf315c39b041fbb658859bddff28db" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41253 } }
I20260929 10:32:20.700706 6980 ts_tablet_manager.cc:1428] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 10:32:20.701452 6961 heartbeater.cc:503] Master 127.20.92.62:36003 was elected leader, sending a full tablet report...
I20260929 10:32:20.742137 6986 raft_consensus.cc:482] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:32:20.742365 6986 raft_consensus.cc:504] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc4d959c27ef47a1a275928825dfb868" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 34523 } } peers { permanent_uuid: "d3760ffaa2bf45fab30b6903b2e2ac8f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 45155 } } peers { permanent_uuid: "f7cf315c39b041fbb658859bddff28db" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41253 } }
I20260929 10:32:20.743762 6986 leader_election.cc:288] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc4d959c27ef47a1a275928825dfb868 (127.20.92.1:34523), f7cf315c39b041fbb658859bddff28db (127.20.92.2:41253)
I20260929 10:32:20.747856 6963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:20.743949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:34523 (local address 127.20.92.3:34497)
0929 10:32:20.744209 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:20.744225 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:20.744321 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:20.744668 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:20.744672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:20.744694 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:32:20.744829 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:20.744836 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.745702 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.745706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:20.746568 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:20.746577 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.747420 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.747424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:20.747523 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:20.747557 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 10:32:20.747645 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 10:32:20.747717 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
I20260929 10:32:20.747856 6974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:20.744103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:34497 (local address 127.20.92.1:34523)
0929 10:32:20.744241 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:20.744248 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:20.744274 (+ 26us) server_negotiation.cc:407] Connection header received
0929 10:32:20.744465 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:20.744469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:20.744537 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:20.744629 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:32:20.744946 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.745589 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.746697 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.747291 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.747423 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:20.747493 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:20.747675 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 10:32:20.747748 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 10:32:20.748100 6977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:20.744197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:35169 (local address 127.20.92.2:41253)
0929 10:32:20.744340 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:20.744345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:20.744640 (+ 295us) server_negotiation.cc:407] Connection header received
0929 10:32:20.744697 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:20.744700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:20.744744 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:20.744834 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:32:20.745141 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.745621 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.746854 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.747310 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.747438 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:20.747517 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:20.747866 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 10:32:20.747953 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 10:32:20.748226 6987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:20.744057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:41253 (local address 127.20.92.3:35169)
0929 10:32:20.744478 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:20.744492 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:20.744618 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:20.744856 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:20.744858 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:20.744873 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:32:20.745023 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:20.745030 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.745808 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.745811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:20.746729 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:20.746740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.747453 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.747457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:20.747720 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:20.747751 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 10:32:20.747851 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 10:32:20.748066 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":139,"threads_started":1}
I20260929 10:32:20.748869 6644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b0a74b87f5647c387a6d1d5281c6d82" candidate_uuid: "d3760ffaa2bf45fab30b6903b2e2ac8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc4d959c27ef47a1a275928825dfb868" is_pre_election: true
I20260929 10:32:20.748883 6779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b0a74b87f5647c387a6d1d5281c6d82" candidate_uuid: "d3760ffaa2bf45fab30b6903b2e2ac8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7cf315c39b041fbb658859bddff28db" is_pre_election: true
I20260929 10:32:20.749215 6779 raft_consensus.cc:2454] T 3b0a74b87f5647c387a6d1d5281c6d82 P f7cf315c39b041fbb658859bddff28db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3760ffaa2bf45fab30b6903b2e2ac8f in term 0.
I20260929 10:32:20.749224 6644 raft_consensus.cc:2454] T 3b0a74b87f5647c387a6d1d5281c6d82 P fc4d959c27ef47a1a275928825dfb868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3760ffaa2bf45fab30b6903b2e2ac8f in term 0.
I20260929 10:32:20.749778 6848 leader_election.cc:302] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: d3760ffaa2bf45fab30b6903b2e2ac8f, f7cf315c39b041fbb658859bddff28db, fc4d959c27ef47a1a275928825dfb868; no voters:
I20260929 10:32:20.750093 6986 raft_consensus.cc:2781] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:32:20.750177 6986 raft_consensus.cc:482] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:32:20.750221 6986 raft_consensus.cc:3037] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:20.751368 6986 raft_consensus.cc:504] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc4d959c27ef47a1a275928825dfb868" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 34523 } } peers { permanent_uuid: "d3760ffaa2bf45fab30b6903b2e2ac8f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 45155 } } peers { permanent_uuid: "f7cf315c39b041fbb658859bddff28db" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41253 } }
I20260929 10:32:20.751693 6986 leader_election.cc:288] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f [CANDIDATE]: Term 1 election: Requested vote from peers fc4d959c27ef47a1a275928825dfb868 (127.20.92.1:34523), f7cf315c39b041fbb658859bddff28db (127.20.92.2:41253)
I20260929 10:32:20.752041 6779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b0a74b87f5647c387a6d1d5281c6d82" candidate_uuid: "d3760ffaa2bf45fab30b6903b2e2ac8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7cf315c39b041fbb658859bddff28db"
I20260929 10:32:20.752024 6644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b0a74b87f5647c387a6d1d5281c6d82" candidate_uuid: "d3760ffaa2bf45fab30b6903b2e2ac8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc4d959c27ef47a1a275928825dfb868"
I20260929 10:32:20.752147 6779 raft_consensus.cc:3037] T 3b0a74b87f5647c387a6d1d5281c6d82 P f7cf315c39b041fbb658859bddff28db [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:20.752195 6644 raft_consensus.cc:3037] T 3b0a74b87f5647c387a6d1d5281c6d82 P fc4d959c27ef47a1a275928825dfb868 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:20.753181 6779 raft_consensus.cc:2454] T 3b0a74b87f5647c387a6d1d5281c6d82 P f7cf315c39b041fbb658859bddff28db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3760ffaa2bf45fab30b6903b2e2ac8f in term 1.
I20260929 10:32:20.753306 6644 raft_consensus.cc:2454] T 3b0a74b87f5647c387a6d1d5281c6d82 P fc4d959c27ef47a1a275928825dfb868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3760ffaa2bf45fab30b6903b2e2ac8f in term 1.
I20260929 10:32:20.753492 6848 leader_election.cc:302] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3760ffaa2bf45fab30b6903b2e2ac8f, f7cf315c39b041fbb658859bddff28db; no voters:
I20260929 10:32:20.753713 6986 raft_consensus.cc:2781] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:32:20.754024 6986 raft_consensus.cc:686] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f [term 1 LEADER]: Becoming Leader. State: Replica: d3760ffaa2bf45fab30b6903b2e2ac8f, State: Running, Role: LEADER
I20260929 10:32:20.754370 6986 consensus_queue.cc:237] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc4d959c27ef47a1a275928825dfb868" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 34523 } } peers { permanent_uuid: "d3760ffaa2bf45fab30b6903b2e2ac8f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 45155 } } peers { permanent_uuid: "f7cf315c39b041fbb658859bddff28db" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41253 } }
I20260929 10:32:20.757882 6497 catalog_manager.cc:5495] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f reported cstate change: term changed from 0 to 1, leader changed from <none> to d3760ffaa2bf45fab30b6903b2e2ac8f (127.20.92.3). New cstate: current_term: 1 leader_uuid: "d3760ffaa2bf45fab30b6903b2e2ac8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc4d959c27ef47a1a275928825dfb868" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 34523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3760ffaa2bf45fab30b6903b2e2ac8f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 45155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7cf315c39b041fbb658859bddff28db" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41253 } health_report { overall_health: UNKNOWN } } }
I20260929 10:32:20.778194 6976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:20.774555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45184 (local address 127.20.92.3:45155)
0929 10:32:20.774684 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:20.774689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:20.774704 (+ 15us) server_negotiation.cc:407] Connection header received
0929 10:32:20.774830 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:20.774832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:20.774886 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:20.774994 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:32:20.775269 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.775785 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.776663 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:20.776880 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:20.776992 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:20.777312 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:32:20.777707 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:32:20.777892 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:20.777993 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 10:32:20.778034 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 10:32:20.784216 6644 raft_consensus.cc:1261] T 3b0a74b87f5647c387a6d1d5281c6d82 P fc4d959c27ef47a1a275928825dfb868 [term 1 FOLLOWER]: Refusing update from remote peer d3760ffaa2bf45fab30b6903b2e2ac8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:32:20.784217 6779 raft_consensus.cc:1261] T 3b0a74b87f5647c387a6d1d5281c6d82 P f7cf315c39b041fbb658859bddff28db [term 1 FOLLOWER]: Refusing update from remote peer d3760ffaa2bf45fab30b6903b2e2ac8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:32:20.785032 6988 consensus_queue.cc:1035] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc4d959c27ef47a1a275928825dfb868" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 34523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:32:20.785223 6986 consensus_queue.cc:1035] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7cf315c39b041fbb658859bddff28db" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 41253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:32:20.793617 6993 mvcc.cc:205] Tried to move back new op lower bound from 7334616845443883008 to 7334616845333061632. Current Snapshot: MvccSnapshot[applied={T|T < 7334616845443883008}]
I20260929 10:32:20.794984 6992 mvcc.cc:205] Tried to move back new op lower bound from 7334616845443883008 to 7334616845333061632. Current Snapshot: MvccSnapshot[applied={T|T < 7334616845443883008}]
I20260929 10:32:20.802249 6498 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:32:20.802438 6498 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:32:20.805130 6498 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2e23eecbad64f3e8c1837658ce718a7: Sending DeleteTablet for 3 replicas of tablet 3b0a74b87f5647c387a6d1d5281c6d82
I20260929 10:32:20.806054 6624 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b0a74b87f5647c387a6d1d5281c6d82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:32:20 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:40048
I20260929 10:32:20.806072 6894 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b0a74b87f5647c387a6d1d5281c6d82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:32:20 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:45170
I20260929 10:32:20.806157 6758 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b0a74b87f5647c387a6d1d5281c6d82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:32:20 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:46812
I20260929 10:32:20.806594 7002 tablet_replica.cc:330] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f: stopping tablet replica
I20260929 10:32:20.806727 7001 tablet_replica.cc:330] T 3b0a74b87f5647c387a6d1d5281c6d82 P fc4d959c27ef47a1a275928825dfb868: stopping tablet replica
I20260929 10:32:20.806977 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 6560
I20260929 10:32:20.807788 7003 tablet_replica.cc:330] T 3b0a74b87f5647c387a6d1d5281c6d82 P f7cf315c39b041fbb658859bddff28db: stopping tablet replica
I20260929 10:32:20.808082 7003 raft_consensus.cc:2229] T 3b0a74b87f5647c387a6d1d5281c6d82 P f7cf315c39b041fbb658859bddff28db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:32:20.808413 7003 raft_consensus.cc:2258] T 3b0a74b87f5647c387a6d1d5281c6d82 P f7cf315c39b041fbb658859bddff28db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:32:20.809084 7002 raft_consensus.cc:2229] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:32:20.809520 7002 raft_consensus.cc:2258] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:32:20.810491 7003 ts_tablet_manager.cc:1905] T 3b0a74b87f5647c387a6d1d5281c6d82 P f7cf315c39b041fbb658859bddff28db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:32:20.810976 7002 ts_tablet_manager.cc:1905] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:32:20.813377 7003 ts_tablet_manager.cc:1918] T 3b0a74b87f5647c387a6d1d5281c6d82 P f7cf315c39b041fbb658859bddff28db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:32:20.813483 7003 log.cc:1196] T 3b0a74b87f5647c387a6d1d5281c6d82 P f7cf315c39b041fbb658859bddff28db: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-1/wal/wals/3b0a74b87f5647c387a6d1d5281c6d82
I20260929 10:32:20.813872 7003 ts_tablet_manager.cc:1939] T 3b0a74b87f5647c387a6d1d5281c6d82 P f7cf315c39b041fbb658859bddff28db: Deleting consensus metadata
I20260929 10:32:20.814373 7002 ts_tablet_manager.cc:1918] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:32:20.814476 7002 log.cc:1196] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677679287822-20848-0/minicluster-data/ts-2/wal/wals/3b0a74b87f5647c387a6d1d5281c6d82
W20260929 10:32:20.814735 6484 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.92.1:34523 (error 108)
I20260929 10:32:20.814810 7002 ts_tablet_manager.cc:1939] T 3b0a74b87f5647c387a6d1d5281c6d82 P d3760ffaa2bf45fab30b6903b2e2ac8f: Deleting consensus metadata
I20260929 10:32:20.815160 6483 catalog_manager.cc:4841] TS f7cf315c39b041fbb658859bddff28db (127.20.92.2:41253): tablet 3b0a74b87f5647c387a6d1d5281c6d82 (table test-table [id=7c9c4677466a4b048e2fa226464ea904]) successfully deleted
I20260929 10:32:20.815598 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 6695
I20260929 10:32:20.816141 6973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:20.815810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:34523 (local address 127.0.0.1:40054)
0929 10:32:20.815954 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:20.816034 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.1:34523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 10:32:20.816320 6484 catalog_manager.cc:4584] TS fc4d959c27ef47a1a275928825dfb868 (127.20.92.1:34523): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3b0a74b87f5647c387a6d1d5281c6d82: Network error: Client connection negotiation failed: client connection to 127.20.92.1:34523: connect: Connection refused (error 111)
I20260929 10:32:20.816543 6485 catalog_manager.cc:4841] TS d3760ffaa2bf45fab30b6903b2e2ac8f (127.20.92.3:45155): tablet 3b0a74b87f5647c387a6d1d5281c6d82 (table test-table [id=7c9c4677466a4b048e2fa226464ea904]) successfully deleted
I20260929 10:32:20.821662 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 6831
I20260929 10:32:20.827453 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 6466
2026-09-29T10:32:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2327 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T10:32:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:32:20Z Disabled control of system clock
I20260929 10:32:20.857992 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:41377
--webserver_interface=127.20.92.62
--webserver_port=0
--builtin_ntp_servers=127.20.92.20:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:41377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 10:32:20.973847 7008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:20.974136 7008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:20.974174 7008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:20.978214 7008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:32:20.978276 7008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:20.978291 7008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:20.978308 7008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:32:20.978323 7008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:32:20.983337 7008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:41377
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:41377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:20.984421 7008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:20.985800 7008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:20.994596 7016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:20.994678 7013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:20.994818 7014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:20.994879 7008 server_base.cc:1034] running on GCE node
I20260929 10:32:20.995507 7008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:20.996747 7008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:20.997912 7008 hybrid_clock.cc:648] HybridClock initialized: now 1790677940997899 us; error 26 us; skew 500 ppm
I20260929 10:32:20.999958 7008 webserver.cc:460] Webserver started at http://127.20.92.62:33811/ using document root <none> and password file <none>
I20260929 10:32:21.000553 7008 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:21.000622 7008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:21.000849 7008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:21.002992 7008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "07a44586f40b4198b23149db5a5aa10f"
format_stamp: "Formatted at 2026-09-29 10:32:20 on dist-test-slave-9cj5"
server_key: "6a2c7ab8c824305729ad442c9d64b1c2"
server_key_iv: "ce2b4a146959fa3e0ac2fca461ed2831"
server_key_version: "encryptionkey@0"
I20260929 10:32:21.003674 7008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "07a44586f40b4198b23149db5a5aa10f"
format_stamp: "Formatted at 2026-09-29 10:32:20 on dist-test-slave-9cj5"
server_key: "6a2c7ab8c824305729ad442c9d64b1c2"
server_key_iv: "ce2b4a146959fa3e0ac2fca461ed2831"
server_key_version: "encryptionkey@0"
I20260929 10:32:21.007395 7008 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 10:32:21.009874 7023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:21.011104 7008 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:32:21.011245 7008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "07a44586f40b4198b23149db5a5aa10f"
format_stamp: "Formatted at 2026-09-29 10:32:20 on dist-test-slave-9cj5"
server_key: "6a2c7ab8c824305729ad442c9d64b1c2"
server_key_iv: "ce2b4a146959fa3e0ac2fca461ed2831"
server_key_version: "encryptionkey@0"
I20260929 10:32:21.011358 7008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:21.057988 7008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:21.058722 7008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:21.058923 7008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:21.066071 7008 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:41377
I20260929 10:32:21.066087 7075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:41377 every 8 connection(s)
I20260929 10:32:21.067008 7008 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:32:21.070165 7076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:21.074708 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 7008
I20260929 10:32:21.074836 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:32:21.075122 20848 external_mini_cluster.cc:1428] Setting key 40065092e20e1a7d03876e06b74e9be8
I20260929 10:32:21.076160 7076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07a44586f40b4198b23149db5a5aa10f: Bootstrap starting.
I20260929 10:32:21.078686 7076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07a44586f40b4198b23149db5a5aa10f: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:21.079511 7076 log.cc:824] T 00000000000000000000000000000000 P 07a44586f40b4198b23149db5a5aa10f: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:21.081444 7076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07a44586f40b4198b23149db5a5aa10f: No bootstrap required, opened a new log
I20260929 10:32:21.084527 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:21.076609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34314 (local address 127.20.92.62:41377)
0929 10:32:21.077289 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:21.077304 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:21.077340 (+ 36us) server_negotiation.cc:407] Connection header received
0929 10:32:21.078089 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:21.078121 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:21.078514 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:21.078846 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:21.079284 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.080301 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.081205 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.081726 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.081948 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:21.082817 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:21.082823 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:21.082845 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:21.083225 (+ 380us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:21.083280 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:21.083299 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:21.083409 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:21.083531 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 10:32:21.083796 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":132,"threads_started":1}
I20260929 10:32:21.085575 7076 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07a44586f40b4198b23149db5a5aa10f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a44586f40b4198b23149db5a5aa10f" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 41377 } }
I20260929 10:32:21.085891 7076 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07a44586f40b4198b23149db5a5aa10f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:21.085963 7076 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07a44586f40b4198b23149db5a5aa10f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07a44586f40b4198b23149db5a5aa10f, State: Initialized, Role: FOLLOWER
I20260929 10:32:21.086462 7076 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07a44586f40b4198b23149db5a5aa10f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a44586f40b4198b23149db5a5aa10f" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 41377 } }
I20260929 10:32:21.086596 7076 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07a44586f40b4198b23149db5a5aa10f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:32:21.086680 7076 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07a44586f40b4198b23149db5a5aa10f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:32:21.086807 7076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07a44586f40b4198b23149db5a5aa10f [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:21.087939 7076 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07a44586f40b4198b23149db5a5aa10f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a44586f40b4198b23149db5a5aa10f" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 41377 } }
I20260929 10:32:21.088313 7076 leader_election.cc:302] T 00000000000000000000000000000000 P 07a44586f40b4198b23149db5a5aa10f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07a44586f40b4198b23149db5a5aa10f; no voters:
I20260929 10:32:21.088639 7076 leader_election.cc:288] T 00000000000000000000000000000000 P 07a44586f40b4198b23149db5a5aa10f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:32:21.088786 7081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07a44586f40b4198b23149db5a5aa10f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:32:21.089025 7081 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07a44586f40b4198b23149db5a5aa10f [term 1 LEADER]: Becoming Leader. State: Replica: 07a44586f40b4198b23149db5a5aa10f, State: Running, Role: LEADER
I20260929 10:32:21.089330 7081 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07a44586f40b4198b23149db5a5aa10f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a44586f40b4198b23149db5a5aa10f" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 41377 } }
I20260929 10:32:21.090009 7076 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07a44586f40b4198b23149db5a5aa10f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:32:21.091158 7083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07a44586f40b4198b23149db5a5aa10f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07a44586f40b4198b23149db5a5aa10f. Latest consensus state: current_term: 1 leader_uuid: "07a44586f40b4198b23149db5a5aa10f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a44586f40b4198b23149db5a5aa10f" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 41377 } } }
I20260929 10:32:21.091324 7083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07a44586f40b4198b23149db5a5aa10f [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:21.091158 7082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07a44586f40b4198b23149db5a5aa10f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07a44586f40b4198b23149db5a5aa10f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a44586f40b4198b23149db5a5aa10f" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 41377 } } }
I20260929 10:32:21.091616 7082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07a44586f40b4198b23149db5a5aa10f [sys.catalog]: This master's current role is: LEADER
W20260929 10:32:21.095103 7097 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 07a44586f40b4198b23149db5a5aa10f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:32:21.095193 7097 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:32:21.095269 7090 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:32:21.096079 7090 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:32:21.101809 7090 catalog_manager.cc:1347] Generated new cluster ID: fbaab6229c9a4c18890aeee3ea4e64df
I20260929 10:32:21.101877 7090 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:32:21.123785 7090 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:32:21.124750 7090 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:32:21.131624 7090 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07a44586f40b4198b23149db5a5aa10f: Generated new TSK 0
I20260929 10:32:21.132308 7090 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:32:21.140106 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:0
--local_ip_for_outbound_sockets=127.20.92.1
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:41377
--builtin_ntp_servers=127.20.92.20:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 10:32:21.254266 7100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:21.254544 7100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:21.254591 7100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:21.258803 7100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:21.258876 7100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:21.258935 7100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:32:21.263963 7100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:41377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:21.265152 7100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:21.266613 7100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:21.276135 7108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:21.276335 7100 server_base.cc:1034] running on GCE node
W20260929 10:32:21.276108 7105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:21.276108 7106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:21.276840 7100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:21.277462 7100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:21.278666 7100 hybrid_clock.cc:648] HybridClock initialized: now 1790677941278600 us; error 74 us; skew 500 ppm
I20260929 10:32:21.280675 7100 webserver.cc:460] Webserver started at http://127.20.92.1:46389/ using document root <none> and password file <none>
I20260929 10:32:21.281301 7100 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:21.281397 7100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:21.281638 7100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:21.283869 7100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-0/data/instance:
uuid: "aa6648a248a54ea39a9d5b5c4fad24ea"
format_stamp: "Formatted at 2026-09-29 10:32:21 on dist-test-slave-9cj5"
server_key: "fcb6c13ad5c3e9bbb1a039853ad1b0e2"
server_key_iv: "9128b153ab55fddcaf5ac8b3f83ecef0"
server_key_version: "encryptionkey@0"
I20260929 10:32:21.284442 7100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance:
uuid: "aa6648a248a54ea39a9d5b5c4fad24ea"
format_stamp: "Formatted at 2026-09-29 10:32:21 on dist-test-slave-9cj5"
server_key: "fcb6c13ad5c3e9bbb1a039853ad1b0e2"
server_key_iv: "9128b153ab55fddcaf5ac8b3f83ecef0"
server_key_version: "encryptionkey@0"
I20260929 10:32:21.288266 7100 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 10:32:21.290975 7114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:21.292433 7100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:32:21.292598 7100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "aa6648a248a54ea39a9d5b5c4fad24ea"
format_stamp: "Formatted at 2026-09-29 10:32:21 on dist-test-slave-9cj5"
server_key: "fcb6c13ad5c3e9bbb1a039853ad1b0e2"
server_key_iv: "9128b153ab55fddcaf5ac8b3f83ecef0"
server_key_version: "encryptionkey@0"
I20260929 10:32:21.292712 7100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:21.309806 7100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:21.310644 7100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:21.310842 7100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:21.311448 7100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:32:21.312352 7100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:32:21.312397 7100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:21.312438 7100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:32:21.312453 7100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:21.321115 7100 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:39085
I20260929 10:32:21.321149 7229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:39085 every 8 connection(s)
I20260929 10:32:21.322029 7100 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
I20260929 10:32:21.326905 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 7100
I20260929 10:32:21.327044 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance
I20260929 10:32:21.327369 20848 external_mini_cluster.cc:1428] Setting key d69ceb10ffe9c3919b8a13af10fb9ac8
I20260929 10:32:21.329465 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:0
--local_ip_for_outbound_sockets=127.20.92.2
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:41377
--builtin_ntp_servers=127.20.92.20:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 10:32:21.332379 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:21.323979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:36897 (local address 127.20.92.62:41377)
0929 10:32:21.324135 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:21.324139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:21.325099 (+ 960us) server_negotiation.cc:407] Connection header received
0929 10:32:21.326560 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:21.326563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:21.326625 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:21.326696 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:21.327925 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.328463 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.330137 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.330348 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.330431 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:21.331802 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:21.331808 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:21.331812 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:21.331827 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:21.331851 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:21.331862 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:21.331942 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:21.332099 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 10:32:21.332169 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 10:32:21.333256 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:21.324325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41377 (local address 127.20.92.1:36897)
0929 10:32:21.324964 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:21.325002 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:21.326358 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:21.326910 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:21.326921 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:21.327037 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:21.327778 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:21.327794 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.328590 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.328593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:21.330003 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:21.330017 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.330513 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.330520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:21.330750 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:21.331476 (+ 726us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:32:21.331478 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:21.331556 (+ 78us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:32:21.331624 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:21.331952 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:21.331959 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:21.331963 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:32:21.332068 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 10:32:21.332293 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":120,"threads_started":1}
I20260929 10:32:21.334583 7230 heartbeater.cc:347] Connected to a master server at 127.20.92.62:41377
I20260929 10:32:21.334810 7230 heartbeater.cc:464] Registering TS with master...
I20260929 10:32:21.335323 7230 heartbeater.cc:511] Master 127.20.92.62:41377 requested a full tablet report, sending...
I20260929 10:32:21.336814 7040 ts_manager.cc:194] Registered new tserver with Master: aa6648a248a54ea39a9d5b5c4fad24ea (127.20.92.1:39085)
I20260929 10:32:21.338021 7040 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.92.1:36897
W20260929 10:32:21.440399 7233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:21.440694 7233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:21.440735 7233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:21.445235 7233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:21.445303 7233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:21.445364 7233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:32:21.450596 7233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:41377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:21.451799 7233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:21.453274 7233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:21.462900 7239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:21.462900 7238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:21.463048 7233 server_base.cc:1034] running on GCE node
W20260929 10:32:21.463235 7241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:21.463624 7233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:21.464228 7233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:21.465441 7233 hybrid_clock.cc:648] HybridClock initialized: now 1790677941465414 us; error 39 us; skew 500 ppm
I20260929 10:32:21.467491 7233 webserver.cc:460] Webserver started at http://127.20.92.2:44449/ using document root <none> and password file <none>
I20260929 10:32:21.468020 7233 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:21.468078 7233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:21.468238 7233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:21.470237 7233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-1/data/instance:
uuid: "ccb320f699f34fb0800ab8107acdb003"
format_stamp: "Formatted at 2026-09-29 10:32:21 on dist-test-slave-9cj5"
server_key: "fc30bb04a291c836e2cc73647519ecc1"
server_key_iv: "c6edd9fdf1b18e0028eceadb9efcf76e"
server_key_version: "encryptionkey@0"
I20260929 10:32:21.470723 7233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance:
uuid: "ccb320f699f34fb0800ab8107acdb003"
format_stamp: "Formatted at 2026-09-29 10:32:21 on dist-test-slave-9cj5"
server_key: "fc30bb04a291c836e2cc73647519ecc1"
server_key_iv: "c6edd9fdf1b18e0028eceadb9efcf76e"
server_key_version: "encryptionkey@0"
I20260929 10:32:21.474283 7233 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:32:21.476737 7247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:21.478017 7233 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:32:21.478165 7233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "ccb320f699f34fb0800ab8107acdb003"
format_stamp: "Formatted at 2026-09-29 10:32:21 on dist-test-slave-9cj5"
server_key: "fc30bb04a291c836e2cc73647519ecc1"
server_key_iv: "c6edd9fdf1b18e0028eceadb9efcf76e"
server_key_version: "encryptionkey@0"
I20260929 10:32:21.478284 7233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:21.507572 7233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:21.508316 7233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:21.508524 7233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:21.509145 7233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:32:21.510138 7233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:32:21.510212 7233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:21.510284 7233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:32:21.510327 7233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:21.519717 7233 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:44835
I20260929 10:32:21.519740 7360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:44835 every 8 connection(s)
I20260929 10:32:21.520612 7233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
I20260929 10:32:21.526674 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 7233
I20260929 10:32:21.526793 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance
I20260929 10:32:21.527130 20848 external_mini_cluster.cc:1428] Setting key d61a912e88bbe21cc8e6594e5f33c6eb
I20260929 10:32:21.529274 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:0
--local_ip_for_outbound_sockets=127.20.92.3
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:41377
--builtin_ntp_servers=127.20.92.20:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 10:32:21.530777 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:21.522615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:44561 (local address 127.20.92.62:41377)
0929 10:32:21.522761 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:21.522765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:21.523819 (+ 1054us) server_negotiation.cc:407] Connection header received
0929 10:32:21.525260 (+ 1441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:21.525264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:21.525323 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:21.525436 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:21.526378 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.527298 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.528627 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.528916 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.529036 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:21.530202 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:21.530209 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:21.530215 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:21.530233 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:21.530259 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:21.530273 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:21.530355 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:21.530494 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 10:32:21.530570 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 10:32:21.531873 7363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:21.522973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41377 (local address 127.20.92.2:44561)
0929 10:32:21.523658 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:21.523700 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:21.525048 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:21.525606 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:21.525621 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:21.525704 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:21.526233 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:21.526246 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.527431 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.527434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:21.528485 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:21.528498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.529070 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.529073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:21.529213 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:21.529931 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:32:21.529932 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:21.529973 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:32:21.530038 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:21.530362 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:21.530368 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:21.530372 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:32:21.530466 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 10:32:21.530802 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":112,"threads_started":1}
I20260929 10:32:21.533041 7361 heartbeater.cc:347] Connected to a master server at 127.20.92.62:41377
I20260929 10:32:21.533317 7361 heartbeater.cc:464] Registering TS with master...
I20260929 10:32:21.533939 7361 heartbeater.cc:511] Master 127.20.92.62:41377 requested a full tablet report, sending...
I20260929 10:32:21.534993 7040 ts_manager.cc:194] Registered new tserver with Master: ccb320f699f34fb0800ab8107acdb003 (127.20.92.2:44835)
I20260929 10:32:21.535557 7040 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.92.2:44561
W20260929 10:32:21.639681 7364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:21.639930 7364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:21.639967 7364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:21.643996 7364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:21.644050 7364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:21.644136 7364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:32:21.649834 7364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:41377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:21.650945 7364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:21.652351 7364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:21.661849 7369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:21.661851 7370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:21.661921 7364 server_base.cc:1034] running on GCE node
W20260929 10:32:21.661849 7372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:21.662441 7364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:21.663038 7364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:21.664223 7364 hybrid_clock.cc:648] HybridClock initialized: now 1790677941664199 us; error 47 us; skew 500 ppm
I20260929 10:32:21.666118 7364 webserver.cc:460] Webserver started at http://127.20.92.3:40465/ using document root <none> and password file <none>
I20260929 10:32:21.666697 7364 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:21.666788 7364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:21.667013 7364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:21.669003 7364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-2/data/instance:
uuid: "036fd96643fd4b71a9aa18a726125068"
format_stamp: "Formatted at 2026-09-29 10:32:21 on dist-test-slave-9cj5"
server_key: "d22310b79602a866f0c92aadc9db1126"
server_key_iv: "e35b33ab28bcb48634b44507010bcc31"
server_key_version: "encryptionkey@0"
I20260929 10:32:21.669564 7364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance:
uuid: "036fd96643fd4b71a9aa18a726125068"
format_stamp: "Formatted at 2026-09-29 10:32:21 on dist-test-slave-9cj5"
server_key: "d22310b79602a866f0c92aadc9db1126"
server_key_iv: "e35b33ab28bcb48634b44507010bcc31"
server_key_version: "encryptionkey@0"
I20260929 10:32:21.673231 7364 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:32:21.675793 7378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:21.677104 7364 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:32:21.677250 7364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "036fd96643fd4b71a9aa18a726125068"
format_stamp: "Formatted at 2026-09-29 10:32:21 on dist-test-slave-9cj5"
server_key: "d22310b79602a866f0c92aadc9db1126"
server_key_iv: "e35b33ab28bcb48634b44507010bcc31"
server_key_version: "encryptionkey@0"
I20260929 10:32:21.677366 7364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:21.709941 7364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:21.710685 7364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:21.710897 7364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:21.711503 7364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:32:21.712474 7364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:32:21.712549 7364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:21.712620 7364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:32:21.712667 7364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:21.721928 7364 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:40517
I20260929 10:32:21.721985 7491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:40517 every 8 connection(s)
I20260929 10:32:21.722930 7364 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:32:21.726454 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 7364
I20260929 10:32:21.726572 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance
I20260929 10:32:21.726867 20848 external_mini_cluster.cc:1428] Setting key f8093a9dbc28824cdae30087e3f13b0c
I20260929 10:32:21.732319 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:21.724767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:53493 (local address 127.20.92.62:41377)
0929 10:32:21.724938 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:21.724942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:21.725981 (+ 1039us) server_negotiation.cc:407] Connection header received
0929 10:32:21.727718 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:21.727722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:21.727778 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:21.727882 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:21.728921 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.729489 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.730661 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.730866 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.730940 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:21.731921 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:21.731924 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:21.731926 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:21.731937 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:21.731955 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:21.731962 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:21.732019 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:21.732162 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 10:32:21.732210 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 10:32:21.733108 7494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:21.725121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:41377 (local address 127.20.92.3:53493)
0929 10:32:21.725839 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:21.725880 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:21.727510 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:21.728085 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:21.728096 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:21.728211 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:21.728752 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:21.728765 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.729617 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.729620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:21.730534 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:21.730543 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.730991 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.730994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:21.731128 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:21.731690 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:32:21.731691 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:21.731727 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:32:21.731786 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:21.732034 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:21.732040 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:21.732044 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:32:21.732137 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 10:32:21.732340 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":114,"threads_started":1}
I20260929 10:32:21.734148 7492 heartbeater.cc:347] Connected to a master server at 127.20.92.62:41377
I20260929 10:32:21.734371 7492 heartbeater.cc:464] Registering TS with master...
I20260929 10:32:21.734870 7492 heartbeater.cc:511] Master 127.20.92.62:41377 requested a full tablet report, sending...
I20260929 10:32:21.735934 7040 ts_manager.cc:194] Registered new tserver with Master: 036fd96643fd4b71a9aa18a726125068 (127.20.92.3:40517)
I20260929 10:32:21.736490 7040 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.92.3:53493
I20260929 10:32:21.740517 20848 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:32:21.746143 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:21.742407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34330 (local address 127.20.92.62:41377)
0929 10:32:21.742576 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:21.742581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:21.742664 (+ 83us) server_negotiation.cc:407] Connection header received
0929 10:32:21.742756 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:21.742759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:21.742794 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:21.742857 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:21.743062 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.743649 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.744487 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.744726 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.744810 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:21.745541 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:21.745544 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:21.745568 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:21.745579 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:21.745597 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:21.745605 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:21.745705 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:21.745903 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 10:32:21.745963 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 10:32:21.748797 7040 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:32:21.751794 7040 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:32:21.766494 7507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:21.761295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39472 (local address 127.20.92.2:44835)
0929 10:32:21.761987 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:21.761999 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:21.762020 (+ 21us) server_negotiation.cc:407] Connection header received
0929 10:32:21.762087 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:21.762091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:21.762244 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:21.762460 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:32:21.762806 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.763579 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.764996 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.765781 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.765878 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:21.766063 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:21.766231 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 10:32:21.766329 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":134,"threads_started":1}
I20260929 10:32:21.766490 7504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:21.760711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:44835 (local address 127.0.0.1:39472)
0929 10:32:21.761323 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:21.761371 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:21.761489 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:21.762473 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:21.762485 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:21.762513 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:32:21.762671 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:21.762680 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.763707 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.763711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:21.764858 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:21.764868 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.765897 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.765901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:21.766027 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:21.766070 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 10:32:21.766188 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 10:32:21.766310 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"spinlock_wait_cycles":896,"thread_start_us":105,"threads_started":1}
I20260929 10:32:21.767258 7506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:21.760518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:40517 (local address 127.0.0.1:38260)
0929 10:32:21.761315 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:21.761361 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:21.761490 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:21.762538 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:21.762541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:21.762559 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:32:21.762670 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:21.762676 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.764265 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.764268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:21.765365 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:21.765375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.766784 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.766789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:21.766895 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:21.766918 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 10:32:21.767022 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 10:32:21.767095 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":4,"thread_start_us":82,"threads_started":1}
I20260929 10:32:21.767566 7503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:21.760969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:39085 (local address 127.0.0.1:35910)
0929 10:32:21.761939 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:21.761952 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:21.762031 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:21.762691 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:21.762694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:21.762713 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:32:21.762801 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:21.762806 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.764301 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.764304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:21.765610 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:21.765621 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.766913 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.766917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:21.767008 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:21.767039 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 10:32:21.767149 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 10:32:21.767237 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"mutex_wait_us":32,"thread_start_us":97,"threads_started":1}
I20260929 10:32:21.767890 7508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:21.761375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35910 (local address 127.20.92.1:39085)
0929 10:32:21.761814 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:21.761826 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:21.762137 (+ 311us) server_negotiation.cc:407] Connection header received
0929 10:32:21.762220 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:21.762225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:21.762424 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:21.762567 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:32:21.762928 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.764167 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.765776 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.766549 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.766688 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:21.766851 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:21.767193 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 10:32:21.767291 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":104,"threads_started":1}
I20260929 10:32:21.767911 7505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:21.760717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38260 (local address 127.20.92.3:40517)
0929 10:32:21.761846 (+ 1129us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:21.761860 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:21.761886 (+ 26us) server_negotiation.cc:407] Connection header received
0929 10:32:21.761968 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:21.761974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:21.762206 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:21.762373 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:32:21.762795 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.764132 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.765505 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.766392 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.766517 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:21.766712 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:21.767457 (+ 745us) server_negotiation.cc:299] Negotiation successful
0929 10:32:21.767565 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":963,"thread_start_us":139,"threads_started":1}
I20260929 10:32:21.770082 7295 tablet_service.cc:1467] Processing CreateTablet for tablet 7ec6e83914d142c2a15e642be722534b (DEFAULT_TABLE table=test-table [id=1254b6571ddd41479d3b0473a86f6ab3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:32:21.770481 7164 tablet_service.cc:1467] Processing CreateTablet for tablet 7ec6e83914d142c2a15e642be722534b (DEFAULT_TABLE table=test-table [id=1254b6571ddd41479d3b0473a86f6ab3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:32:21.770723 7426 tablet_service.cc:1467] Processing CreateTablet for tablet 7ec6e83914d142c2a15e642be722534b (DEFAULT_TABLE table=test-table [id=1254b6571ddd41479d3b0473a86f6ab3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:32:21.771230 7295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ec6e83914d142c2a15e642be722534b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:21.771595 7164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ec6e83914d142c2a15e642be722534b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:21.771662 7426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ec6e83914d142c2a15e642be722534b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:21.778312 7509 tablet_bootstrap.cc:492] T 7ec6e83914d142c2a15e642be722534b P aa6648a248a54ea39a9d5b5c4fad24ea: Bootstrap starting.
I20260929 10:32:21.779495 7510 tablet_bootstrap.cc:492] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068: Bootstrap starting.
I20260929 10:32:21.780496 7509 tablet_bootstrap.cc:654] T 7ec6e83914d142c2a15e642be722534b P aa6648a248a54ea39a9d5b5c4fad24ea: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:21.781313 7509 log.cc:824] T 7ec6e83914d142c2a15e642be722534b P aa6648a248a54ea39a9d5b5c4fad24ea: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:21.781363 7511 tablet_bootstrap.cc:492] T 7ec6e83914d142c2a15e642be722534b P ccb320f699f34fb0800ab8107acdb003: Bootstrap starting.
I20260929 10:32:21.782171 7510 tablet_bootstrap.cc:654] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:21.782974 7510 log.cc:824] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:21.783372 7511 tablet_bootstrap.cc:654] T 7ec6e83914d142c2a15e642be722534b P ccb320f699f34fb0800ab8107acdb003: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:21.783551 7509 tablet_bootstrap.cc:492] T 7ec6e83914d142c2a15e642be722534b P aa6648a248a54ea39a9d5b5c4fad24ea: No bootstrap required, opened a new log
I20260929 10:32:21.783738 7509 ts_tablet_manager.cc:1397] T 7ec6e83914d142c2a15e642be722534b P aa6648a248a54ea39a9d5b5c4fad24ea: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 10:32:21.784114 7511 log.cc:824] T 7ec6e83914d142c2a15e642be722534b P ccb320f699f34fb0800ab8107acdb003: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:21.784897 7510 tablet_bootstrap.cc:492] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068: No bootstrap required, opened a new log
I20260929 10:32:21.785085 7510 ts_tablet_manager.cc:1397] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 10:32:21.785840 7511 tablet_bootstrap.cc:492] T 7ec6e83914d142c2a15e642be722534b P ccb320f699f34fb0800ab8107acdb003: No bootstrap required, opened a new log
I20260929 10:32:21.786015 7511 ts_tablet_manager.cc:1397] T 7ec6e83914d142c2a15e642be722534b P ccb320f699f34fb0800ab8107acdb003: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 10:32:21.788204 7509 raft_consensus.cc:348] T 7ec6e83914d142c2a15e642be722534b P aa6648a248a54ea39a9d5b5c4fad24ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036fd96643fd4b71a9aa18a726125068" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40517 } } peers { permanent_uuid: "ccb320f699f34fb0800ab8107acdb003" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44835 } } peers { permanent_uuid: "aa6648a248a54ea39a9d5b5c4fad24ea" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39085 } }
I20260929 10:32:21.788491 7509 raft_consensus.cc:374] T 7ec6e83914d142c2a15e642be722534b P aa6648a248a54ea39a9d5b5c4fad24ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:21.788550 7509 raft_consensus.cc:729] T 7ec6e83914d142c2a15e642be722534b P aa6648a248a54ea39a9d5b5c4fad24ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa6648a248a54ea39a9d5b5c4fad24ea, State: Initialized, Role: FOLLOWER
I20260929 10:32:21.789031 7509 consensus_queue.cc:260] T 7ec6e83914d142c2a15e642be722534b P aa6648a248a54ea39a9d5b5c4fad24ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036fd96643fd4b71a9aa18a726125068" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40517 } } peers { permanent_uuid: "ccb320f699f34fb0800ab8107acdb003" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44835 } } peers { permanent_uuid: "aa6648a248a54ea39a9d5b5c4fad24ea" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39085 } }
I20260929 10:32:21.789134 7511 raft_consensus.cc:348] T 7ec6e83914d142c2a15e642be722534b P ccb320f699f34fb0800ab8107acdb003 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036fd96643fd4b71a9aa18a726125068" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40517 } } peers { permanent_uuid: "ccb320f699f34fb0800ab8107acdb003" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44835 } } peers { permanent_uuid: "aa6648a248a54ea39a9d5b5c4fad24ea" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39085 } }
I20260929 10:32:21.789389 7511 raft_consensus.cc:374] T 7ec6e83914d142c2a15e642be722534b P ccb320f699f34fb0800ab8107acdb003 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:21.789237 7510 raft_consensus.cc:348] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036fd96643fd4b71a9aa18a726125068" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40517 } } peers { permanent_uuid: "ccb320f699f34fb0800ab8107acdb003" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44835 } } peers { permanent_uuid: "aa6648a248a54ea39a9d5b5c4fad24ea" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39085 } }
I20260929 10:32:21.789433 7511 raft_consensus.cc:729] T 7ec6e83914d142c2a15e642be722534b P ccb320f699f34fb0800ab8107acdb003 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb320f699f34fb0800ab8107acdb003, State: Initialized, Role: FOLLOWER
I20260929 10:32:21.789467 7510 raft_consensus.cc:374] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:21.789521 7510 raft_consensus.cc:729] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 036fd96643fd4b71a9aa18a726125068, State: Initialized, Role: FOLLOWER
I20260929 10:32:21.789861 7511 consensus_queue.cc:260] T 7ec6e83914d142c2a15e642be722534b P ccb320f699f34fb0800ab8107acdb003 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036fd96643fd4b71a9aa18a726125068" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40517 } } peers { permanent_uuid: "ccb320f699f34fb0800ab8107acdb003" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44835 } } peers { permanent_uuid: "aa6648a248a54ea39a9d5b5c4fad24ea" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39085 } }
I20260929 10:32:21.789944 7510 consensus_queue.cc:260] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036fd96643fd4b71a9aa18a726125068" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40517 } } peers { permanent_uuid: "ccb320f699f34fb0800ab8107acdb003" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44835 } } peers { permanent_uuid: "aa6648a248a54ea39a9d5b5c4fad24ea" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39085 } }
I20260929 10:32:21.790148 7509 ts_tablet_manager.cc:1428] T 7ec6e83914d142c2a15e642be722534b P aa6648a248a54ea39a9d5b5c4fad24ea: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 10:32:21.790562 7230 heartbeater.cc:503] Master 127.20.92.62:41377 was elected leader, sending a full tablet report...
I20260929 10:32:21.790872 7510 ts_tablet_manager.cc:1428] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 10:32:21.790993 7511 ts_tablet_manager.cc:1428] T 7ec6e83914d142c2a15e642be722534b P ccb320f699f34fb0800ab8107acdb003: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 10:32:21.791167 7492 heartbeater.cc:503] Master 127.20.92.62:41377 was elected leader, sending a full tablet report...
I20260929 10:32:21.791314 7361 heartbeater.cc:503] Master 127.20.92.62:41377 was elected leader, sending a full tablet report...
W20260929 10:32:21.823237 7231 tablet.cc:2369] T 7ec6e83914d142c2a15e642be722534b P aa6648a248a54ea39a9d5b5c4fad24ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:32:21.946609 7517 raft_consensus.cc:482] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:32:21.946862 7517 raft_consensus.cc:504] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036fd96643fd4b71a9aa18a726125068" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40517 } } peers { permanent_uuid: "ccb320f699f34fb0800ab8107acdb003" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44835 } } peers { permanent_uuid: "aa6648a248a54ea39a9d5b5c4fad24ea" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39085 } }
I20260929 10:32:21.948167 7517 leader_election.cc:288] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccb320f699f34fb0800ab8107acdb003 (127.20.92.2:44835), aa6648a248a54ea39a9d5b5c4fad24ea (127.20.92.1:39085)
I20260929 10:32:21.953289 7507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:21.948581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:35145 (local address 127.20.92.2:44835)
0929 10:32:21.948735 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:21.948740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:21.948755 (+ 15us) server_negotiation.cc:407] Connection header received
0929 10:32:21.948902 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:21.948910 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:21.948955 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:21.949046 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:32:21.949342 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.950065 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.951761 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.952382 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.952493 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:21.952555 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:21.953077 (+ 522us) server_negotiation.cc:299] Negotiation successful
0929 10:32:21.953158 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 10:32:21.953310 7494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:21.948437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:44835 (local address 127.20.92.3:35145)
0929 10:32:21.948650 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:21.948668 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:21.948768 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:21.949073 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:21.949077 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:21.949094 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:32:21.949226 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:21.949233 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.950207 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.950211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:21.951615 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:21.951626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.952643 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.952647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:21.952857 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:21.952933 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 10:32:21.953016 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 10:32:21.953150 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":148}
I20260929 10:32:21.953289 7508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:21.948586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:56447 (local address 127.20.92.1:39085)
0929 10:32:21.948757 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:21.948761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:21.948993 (+ 232us) server_negotiation.cc:407] Connection header received
0929 10:32:21.949138 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:21.949142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:21.949196 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:21.949286 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:32:21.949577 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.950246 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.951761 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.952382 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.952492 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:21.952554 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:21.953077 (+ 523us) server_negotiation.cc:299] Negotiation successful
0929 10:32:21.953158 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 10:32:21.953318 7518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:21.948467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:39085 (local address 127.20.92.3:56447)
0929 10:32:21.948889 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:21.948900 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:21.949028 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:21.949297 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:21.949300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:21.949315 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:32:21.949450 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:21.949456 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.950364 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.950368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:21.951619 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:21.951626 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:21.952643 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:21.952647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:21.952882 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:21.952943 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 10:32:21.953016 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 10:32:21.953138 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":214,"thread_start_us":118,"threads_started":1}
I20260929 10:32:21.954035 7315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec6e83914d142c2a15e642be722534b" candidate_uuid: "036fd96643fd4b71a9aa18a726125068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb320f699f34fb0800ab8107acdb003" is_pre_election: true
I20260929 10:32:21.954284 7184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec6e83914d142c2a15e642be722534b" candidate_uuid: "036fd96643fd4b71a9aa18a726125068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa6648a248a54ea39a9d5b5c4fad24ea" is_pre_election: true
I20260929 10:32:21.954385 7315 raft_consensus.cc:2454] T 7ec6e83914d142c2a15e642be722534b P ccb320f699f34fb0800ab8107acdb003 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 036fd96643fd4b71a9aa18a726125068 in term 0.
I20260929 10:32:21.954561 7184 raft_consensus.cc:2454] T 7ec6e83914d142c2a15e642be722534b P aa6648a248a54ea39a9d5b5c4fad24ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 036fd96643fd4b71a9aa18a726125068 in term 0.
I20260929 10:32:21.954988 7382 leader_election.cc:302] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 036fd96643fd4b71a9aa18a726125068, ccb320f699f34fb0800ab8107acdb003; no voters:
I20260929 10:32:21.955255 7517 raft_consensus.cc:2781] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:32:21.955335 7517 raft_consensus.cc:482] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:32:21.955376 7517 raft_consensus.cc:3037] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:21.956357 7517 raft_consensus.cc:504] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036fd96643fd4b71a9aa18a726125068" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40517 } } peers { permanent_uuid: "ccb320f699f34fb0800ab8107acdb003" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44835 } } peers { permanent_uuid: "aa6648a248a54ea39a9d5b5c4fad24ea" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39085 } }
I20260929 10:32:21.956704 7517 leader_election.cc:288] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068 [CANDIDATE]: Term 1 election: Requested vote from peers ccb320f699f34fb0800ab8107acdb003 (127.20.92.2:44835), aa6648a248a54ea39a9d5b5c4fad24ea (127.20.92.1:39085)
I20260929 10:32:21.957047 7315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec6e83914d142c2a15e642be722534b" candidate_uuid: "036fd96643fd4b71a9aa18a726125068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb320f699f34fb0800ab8107acdb003"
I20260929 10:32:21.957176 7315 raft_consensus.cc:3037] T 7ec6e83914d142c2a15e642be722534b P ccb320f699f34fb0800ab8107acdb003 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:21.957180 7184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec6e83914d142c2a15e642be722534b" candidate_uuid: "036fd96643fd4b71a9aa18a726125068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa6648a248a54ea39a9d5b5c4fad24ea"
I20260929 10:32:21.957319 7184 raft_consensus.cc:3037] T 7ec6e83914d142c2a15e642be722534b P aa6648a248a54ea39a9d5b5c4fad24ea [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:21.958269 7315 raft_consensus.cc:2454] T 7ec6e83914d142c2a15e642be722534b P ccb320f699f34fb0800ab8107acdb003 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 036fd96643fd4b71a9aa18a726125068 in term 1.
I20260929 10:32:21.958492 7184 raft_consensus.cc:2454] T 7ec6e83914d142c2a15e642be722534b P aa6648a248a54ea39a9d5b5c4fad24ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 036fd96643fd4b71a9aa18a726125068 in term 1.
I20260929 10:32:21.958601 7382 leader_election.cc:302] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 036fd96643fd4b71a9aa18a726125068, ccb320f699f34fb0800ab8107acdb003; no voters:
I20260929 10:32:21.958777 7517 raft_consensus.cc:2781] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:32:21.958989 7517 raft_consensus.cc:686] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068 [term 1 LEADER]: Becoming Leader. State: Replica: 036fd96643fd4b71a9aa18a726125068, State: Running, Role: LEADER
I20260929 10:32:21.959303 7517 consensus_queue.cc:237] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036fd96643fd4b71a9aa18a726125068" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40517 } } peers { permanent_uuid: "ccb320f699f34fb0800ab8107acdb003" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44835 } } peers { permanent_uuid: "aa6648a248a54ea39a9d5b5c4fad24ea" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39085 } }
I20260929 10:32:21.962438 7040 catalog_manager.cc:5495] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068 reported cstate change: term changed from 0 to 1, leader changed from <none> to 036fd96643fd4b71a9aa18a726125068 (127.20.92.3). New cstate: current_term: 1 leader_uuid: "036fd96643fd4b71a9aa18a726125068" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036fd96643fd4b71a9aa18a726125068" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccb320f699f34fb0800ab8107acdb003" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa6648a248a54ea39a9d5b5c4fad24ea" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39085 } health_report { overall_health: UNKNOWN } } }
W20260929 10:32:21.973990 7493 tablet.cc:2369] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:32:22.004657 7505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:22.001035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38264 (local address 127.20.92.3:40517)
0929 10:32:22.001176 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:22.001182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:22.001200 (+ 18us) server_negotiation.cc:407] Connection header received
0929 10:32:22.001283 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:22.001286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:22.001339 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:22.001451 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:32:22.001701 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:22.002258 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:22.003151 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:22.003368 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:22.003479 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:22.003798 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:32:22.004203 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:32:22.004362 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:22.004453 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 10:32:22.004488 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 10:32:22.010574 7315 raft_consensus.cc:1261] T 7ec6e83914d142c2a15e642be722534b P ccb320f699f34fb0800ab8107acdb003 [term 1 FOLLOWER]: Refusing update from remote peer 036fd96643fd4b71a9aa18a726125068: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:32:22.010689 7184 raft_consensus.cc:1261] T 7ec6e83914d142c2a15e642be722534b P aa6648a248a54ea39a9d5b5c4fad24ea [term 1 FOLLOWER]: Refusing update from remote peer 036fd96643fd4b71a9aa18a726125068: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:32:22.011392 7519 consensus_queue.cc:1035] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccb320f699f34fb0800ab8107acdb003" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 44835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:32:22.011601 7517 consensus_queue.cc:1035] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa6648a248a54ea39a9d5b5c4fad24ea" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:32:22.018380 7522 mvcc.cc:205] Tried to move back new op lower bound from 7334616850468139008 to 7334616850267725824. Current Snapshot: MvccSnapshot[applied={T|T < 7334616850468139008}]
I20260929 10:32:22.019192 7524 mvcc.cc:205] Tried to move back new op lower bound from 7334616850468139008 to 7334616850267725824. Current Snapshot: MvccSnapshot[applied={T|T < 7334616850468139008}]
I20260929 10:32:22.020305 7523 mvcc.cc:205] Tried to move back new op lower bound from 7334616850468139008 to 7334616850267725824. Current Snapshot: MvccSnapshot[applied={T|T < 7334616850468139008}]
I20260929 10:32:22.026999 7038 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:32:22.027199 7038 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:32:22.029433 7038 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07a44586f40b4198b23149db5a5aa10f: Sending DeleteTablet for 3 replicas of tablet 7ec6e83914d142c2a15e642be722534b
I20260929 10:32:22.030191 7426 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ec6e83914d142c2a15e642be722534b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:32:22 UTC) from {username='slave'} at 127.0.0.1:38260
I20260929 10:32:22.030277 7295 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ec6e83914d142c2a15e642be722534b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:32:22 UTC) from {username='slave'} at 127.0.0.1:39472
I20260929 10:32:22.030721 7532 tablet_replica.cc:330] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068: stopping tablet replica
I20260929 10:32:22.030699 7164 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ec6e83914d142c2a15e642be722534b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:32:22 UTC) from {username='slave'} at 127.0.0.1:35910
I20260929 10:32:22.031109 7532 raft_consensus.cc:2229] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:32:22.031270 7533 tablet_replica.cc:330] T 7ec6e83914d142c2a15e642be722534b P ccb320f699f34fb0800ab8107acdb003: stopping tablet replica
I20260929 10:32:22.031535 7533 raft_consensus.cc:2229] T 7ec6e83914d142c2a15e642be722534b P ccb320f699f34fb0800ab8107acdb003 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:32:22.031633 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 7100
I20260929 10:32:22.031571 7532 raft_consensus.cc:2258] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:32:22.031847 7533 raft_consensus.cc:2258] T 7ec6e83914d142c2a15e642be722534b P ccb320f699f34fb0800ab8107acdb003 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:32:22.033908 7533 ts_tablet_manager.cc:1905] T 7ec6e83914d142c2a15e642be722534b P ccb320f699f34fb0800ab8107acdb003: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:32:22.036598 7532 ts_tablet_manager.cc:1905] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:32:22.036782 7533 ts_tablet_manager.cc:1918] T 7ec6e83914d142c2a15e642be722534b P ccb320f699f34fb0800ab8107acdb003: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:32:22.036859 7533 log.cc:1196] T 7ec6e83914d142c2a15e642be722534b P ccb320f699f34fb0800ab8107acdb003: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-1/wal/wals/7ec6e83914d142c2a15e642be722534b
I20260929 10:32:22.037194 7533 ts_tablet_manager.cc:1939] T 7ec6e83914d142c2a15e642be722534b P ccb320f699f34fb0800ab8107acdb003: Deleting consensus metadata
I20260929 10:32:22.038550 7027 catalog_manager.cc:4841] TS ccb320f699f34fb0800ab8107acdb003 (127.20.92.2:44835): tablet 7ec6e83914d142c2a15e642be722534b (table test-table [id=1254b6571ddd41479d3b0473a86f6ab3]) successfully deleted
I20260929 10:32:22.039289 7532 ts_tablet_manager.cc:1918] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:32:22.039408 7532 log.cc:1196] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677679287822-20848-0/minicluster-data/ts-2/wal/wals/7ec6e83914d142c2a15e642be722534b
I20260929 10:32:22.039683 7532 ts_tablet_manager.cc:1939] T 7ec6e83914d142c2a15e642be722534b P 036fd96643fd4b71a9aa18a726125068: Deleting consensus metadata
I20260929 10:32:22.040679 7026 catalog_manager.cc:4841] TS 036fd96643fd4b71a9aa18a726125068 (127.20.92.3:40517): tablet 7ec6e83914d142c2a15e642be722534b (table test-table [id=1254b6571ddd41479d3b0473a86f6ab3]) successfully deleted
W20260929 10:32:22.042495 7024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.92.1:39085 (error 108)
I20260929 10:32:22.042613 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 7233
I20260929 10:32:22.044241 7503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:22.043911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:39085 (local address 127.0.0.1:35914)
0929 10:32:22.044090 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:22.044158 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.1:39085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260929 10:32:22.044404 7024 catalog_manager.cc:4584] TS aa6648a248a54ea39a9d5b5c4fad24ea (127.20.92.1:39085): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7ec6e83914d142c2a15e642be722534b: Network error: Client connection negotiation failed: client connection to 127.20.92.1:39085: connect: Connection refused (error 111)
I20260929 10:32:22.048768 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 7364
I20260929 10:32:22.054311 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 7008
2026-09-29T10:32:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1225 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:32:22 dist-test-slave-9cj5 krb5kdc[7538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:32:22 dist-test-slave-9cj5 krb5kdc[7538](info): set up 2 sockets
Sep 29 10:32:22 dist-test-slave-9cj5 krb5kdc[7538](info): commencing operation
krb5kdc: starting...
W20260929 10:32:23.098495 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:32:23 dist-test-slave-9cj5 krb5kdc[7538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:32:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:32:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
I20260929 10:32:23.225044 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:43611
--webserver_interface=127.20.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:40833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:43611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:32:23.337620 7551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:23.337922 7551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:23.337970 7551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:23.342188 7551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:32:23.342252 7551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:23.342272 7551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:23.342289 7551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:32:23.342305 7551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:32:23.347707 7551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:40833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:43611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:43611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:23.348821 7551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:23.349884 7551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:23.358579 7556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:23.358672 7551 server_base.cc:1034] running on GCE node
W20260929 10:32:23.358585 7559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:23.358585 7557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:23.359442 7551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:23.360606 7551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:23.361829 7551 hybrid_clock.cc:648] HybridClock initialized: now 1790677943361807 us; error 34 us; skew 500 ppm
Sep 29 10:32:23 dist-test-slave-9cj5 krb5kdc[7538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677943, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:23.364568 7551 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:32:23.365417 7551 webserver.cc:460] Webserver started at http://127.20.92.62:45753/ using document root <none> and password file <none>
I20260929 10:32:23.366008 7551 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:23.366070 7551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:23.366235 7551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:23.368249 7551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "e15c6948556d4d80a30a9d03af28f299"
format_stamp: "Formatted at 2026-09-29 10:32:23 on dist-test-slave-9cj5"
server_key: "cd0590c07ac3c8961037bc543c6c6dd8"
server_key_iv: "19f56ea42c533a5c560e5b144f205d53"
server_key_version: "encryptionkey@0"
I20260929 10:32:23.368746 7551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "e15c6948556d4d80a30a9d03af28f299"
format_stamp: "Formatted at 2026-09-29 10:32:23 on dist-test-slave-9cj5"
server_key: "cd0590c07ac3c8961037bc543c6c6dd8"
server_key_iv: "19f56ea42c533a5c560e5b144f205d53"
server_key_version: "encryptionkey@0"
I20260929 10:32:23.372414 7551 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 10:32:23.374797 7566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:23.376083 7551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:32:23.376181 7551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "e15c6948556d4d80a30a9d03af28f299"
format_stamp: "Formatted at 2026-09-29 10:32:23 on dist-test-slave-9cj5"
server_key: "cd0590c07ac3c8961037bc543c6c6dd8"
server_key_iv: "19f56ea42c533a5c560e5b144f205d53"
server_key_version: "encryptionkey@0"
I20260929 10:32:23.376268 7551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:23.389429 7551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:23.392608 7551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:23.392758 7551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:23.399729 7551 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:43611
I20260929 10:32:23.399755 7619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:43611 every 8 connection(s)
I20260929 10:32:23.400643 7551 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:32:23.401636 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 7551
I20260929 10:32:23.401808 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:32:23.402143 20848 external_mini_cluster.cc:1428] Setting key e72fbaea50e9e2bc3a1d967e164647f2
I20260929 10:32:23.403932 7620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:23.410274 7620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e15c6948556d4d80a30a9d03af28f299: Bootstrap starting.
Sep 29 10:32:23 dist-test-slave-9cj5 krb5kdc[7538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:32:23.412838 7620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e15c6948556d4d80a30a9d03af28f299: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:23.413698 7620 log.cc:824] T 00000000000000000000000000000000 P e15c6948556d4d80a30a9d03af28f299: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:23.415620 7620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e15c6948556d4d80a30a9d03af28f299: No bootstrap required, opened a new log
I20260929 10:32:23.417268 7623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:23.403601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42748 (local address 127.20.92.62:43611)
0929 10:32:23.404093 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:23.404110 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:23.404147 (+ 37us) server_negotiation.cc:407] Connection header received
0929 10:32:23.404879 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:23.404896 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:23.405236 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:23.405529 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:23.406436 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:23.407595 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:23.408752 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:23.409248 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:23.409902 (+ 654us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:23.412352 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:23.412358 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:23.412370 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:23.412399 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:23.414282 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:23.414771 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:23.414776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:23.414780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:23.414891 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:23.415186 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:23.415190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:23.415192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:23.415586 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:23.415786 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:23.416084 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 10:32:23.416370 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":146,"threads_started":1}
I20260929 10:32:23.418934 7620 raft_consensus.cc:348] T 00000000000000000000000000000000 P e15c6948556d4d80a30a9d03af28f299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15c6948556d4d80a30a9d03af28f299" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 43611 } }
I20260929 10:32:23.419198 7620 raft_consensus.cc:374] T 00000000000000000000000000000000 P e15c6948556d4d80a30a9d03af28f299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:23.419255 7620 raft_consensus.cc:729] T 00000000000000000000000000000000 P e15c6948556d4d80a30a9d03af28f299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e15c6948556d4d80a30a9d03af28f299, State: Initialized, Role: FOLLOWER
I20260929 10:32:23.419706 7620 consensus_queue.cc:260] T 00000000000000000000000000000000 P e15c6948556d4d80a30a9d03af28f299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15c6948556d4d80a30a9d03af28f299" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 43611 } }
I20260929 10:32:23.419835 7620 raft_consensus.cc:388] T 00000000000000000000000000000000 P e15c6948556d4d80a30a9d03af28f299 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:32:23.419876 7620 raft_consensus.cc:482] T 00000000000000000000000000000000 P e15c6948556d4d80a30a9d03af28f299 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:32:23.419946 7620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e15c6948556d4d80a30a9d03af28f299 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:23.420815 7620 raft_consensus.cc:504] T 00000000000000000000000000000000 P e15c6948556d4d80a30a9d03af28f299 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15c6948556d4d80a30a9d03af28f299" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 43611 } }
I20260929 10:32:23.421186 7620 leader_election.cc:302] T 00000000000000000000000000000000 P e15c6948556d4d80a30a9d03af28f299 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e15c6948556d4d80a30a9d03af28f299; no voters:
I20260929 10:32:23.421578 7620 leader_election.cc:288] T 00000000000000000000000000000000 P e15c6948556d4d80a30a9d03af28f299 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:32:23.421656 7625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e15c6948556d4d80a30a9d03af28f299 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:32:23.421875 7625 raft_consensus.cc:686] T 00000000000000000000000000000000 P e15c6948556d4d80a30a9d03af28f299 [term 1 LEADER]: Becoming Leader. State: Replica: e15c6948556d4d80a30a9d03af28f299, State: Running, Role: LEADER
I20260929 10:32:23.422178 7625 consensus_queue.cc:237] T 00000000000000000000000000000000 P e15c6948556d4d80a30a9d03af28f299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15c6948556d4d80a30a9d03af28f299" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 43611 } }
I20260929 10:32:23.422876 7620 sys_catalog.cc:564] T 00000000000000000000000000000000 P e15c6948556d4d80a30a9d03af28f299 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:32:23.423934 7627 sys_catalog.cc:455] T 00000000000000000000000000000000 P e15c6948556d4d80a30a9d03af28f299 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e15c6948556d4d80a30a9d03af28f299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15c6948556d4d80a30a9d03af28f299" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 43611 } } }
I20260929 10:32:23.423947 7626 sys_catalog.cc:455] T 00000000000000000000000000000000 P e15c6948556d4d80a30a9d03af28f299 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e15c6948556d4d80a30a9d03af28f299. Latest consensus state: current_term: 1 leader_uuid: "e15c6948556d4d80a30a9d03af28f299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15c6948556d4d80a30a9d03af28f299" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 43611 } } }
I20260929 10:32:23.424271 7626 sys_catalog.cc:458] T 00000000000000000000000000000000 P e15c6948556d4d80a30a9d03af28f299 [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:23.424304 7627 sys_catalog.cc:458] T 00000000000000000000000000000000 P e15c6948556d4d80a30a9d03af28f299 [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:23.425151 7634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:32:23.427780 7634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:32:23.433617 7634 catalog_manager.cc:1347] Generated new cluster ID: 0ed55b06b1364b9385b0d42249f4fcac
I20260929 10:32:23.433722 7634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:32:23.454120 7634 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:32:23.455428 7634 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:32:23.466473 7634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e15c6948556d4d80a30a9d03af28f299: Generated new TSK 0
I20260929 10:32:23.467294 7634 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
Entry for principal kudu/127.20.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab.
I20260929 10:32:23.508805 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:0
--local_ip_for_outbound_sockets=127.20.92.1
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:43611
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:40833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:32:23.623976 7646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:23.624255 7646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:23.624300 7646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:23.628453 7646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:23.628517 7646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:23.628580 7646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:32:23.634393 7646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:40833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.1.keytab
--principal=kudu/127.20.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:43611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:23.635440 7646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:23.636425 7646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:23.646222 7651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:23.646282 7646 server_base.cc:1034] running on GCE node
W20260929 10:32:23.646224 7654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:23.646232 7652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:23.646816 7646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:23.647447 7646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:23.648612 7646 hybrid_clock.cc:648] HybridClock initialized: now 1790677943648590 us; error 33 us; skew 500 ppm
Sep 29 10:32:23 dist-test-slave-9cj5 krb5kdc[7538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677943, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:23.651315 7646 init.cc:377] Logged in from keytab as kudu/127.20.92.1@KRBTEST.COM (short username kudu)
I20260929 10:32:23.652222 7646 webserver.cc:460] Webserver started at http://127.20.92.1:33667/ using document root <none> and password file <none>
I20260929 10:32:23.652781 7646 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:23.652838 7646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:23.653066 7646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:23.655115 7646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-0/data/instance:
uuid: "ba260a5cf7094549a6172b23aafb01b8"
format_stamp: "Formatted at 2026-09-29 10:32:23 on dist-test-slave-9cj5"
server_key: "084810cbce1855d724e4a2559090194c"
server_key_iv: "8375d5801c7b9633cb2b648a3d08c673"
server_key_version: "encryptionkey@0"
I20260929 10:32:23.655675 7646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance:
uuid: "ba260a5cf7094549a6172b23aafb01b8"
format_stamp: "Formatted at 2026-09-29 10:32:23 on dist-test-slave-9cj5"
server_key: "084810cbce1855d724e4a2559090194c"
server_key_iv: "8375d5801c7b9633cb2b648a3d08c673"
server_key_version: "encryptionkey@0"
I20260929 10:32:23.659375 7646 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 10:32:23.661804 7661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:23.663003 7646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 10:32:23.663118 7646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "ba260a5cf7094549a6172b23aafb01b8"
format_stamp: "Formatted at 2026-09-29 10:32:23 on dist-test-slave-9cj5"
server_key: "084810cbce1855d724e4a2559090194c"
server_key_iv: "8375d5801c7b9633cb2b648a3d08c673"
server_key_version: "encryptionkey@0"
I20260929 10:32:23.663244 7646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:23.698195 7646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:23.701334 7646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:23.701551 7646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:23.702206 7646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:32:23.703076 7646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:32:23.703131 7646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:23.703198 7646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:32:23.703230 7646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:23.712219 7646 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:39017
I20260929 10:32:23.712241 7774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:39017 every 8 connection(s)
I20260929 10:32:23.713124 7646 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
I20260929 10:32:23.716866 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 7646
I20260929 10:32:23.717012 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance
I20260929 10:32:23.717322 20848 external_mini_cluster.cc:1428] Setting key 22623ae1e4327ffd0ece887fbaba3366
Sep 29 10:32:23 dist-test-slave-9cj5 krb5kdc[7538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677943, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.1@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.92.2@KRBTEST.COM; defaulting to no policy
I20260929 10:32:23.727566 7623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:23.714982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:44499 (local address 127.20.92.62:43611)
0929 10:32:23.715122 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:23.715127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:23.715917 (+ 790us) server_negotiation.cc:407] Connection header received
0929 10:32:23.717400 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:23.717404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:23.717464 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:23.717542 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:23.718905 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:23.719439 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:23.720576 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:23.720845 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:23.720962 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:23.723697 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:23.723700 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:23.723703 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:23.723735 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:23.725339 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:23.725902 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:23.725905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:23.725907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:23.726005 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:23.726400 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:23.726403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:23.726405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:23.726595 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:23.726710 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:23.727223 (+ 513us) server_negotiation.cc:299] Negotiation successful
0929 10:32:23.727390 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 10:32:23.728363 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:23.715271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:43611 (local address 127.20.92.1:44499)
0929 10:32:23.715732 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:23.715775 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:23.717189 (+ 1414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:23.717735 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:23.717746 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:23.718194 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:23.718746 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:23.718768 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:23.719571 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:23.719575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:23.720451 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:23.720460 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:23.721006 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:23.721010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:23.721188 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:23.721849 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:32:23.721865 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:23.723458 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:23.725476 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:23.725482 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:23.725498 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:23.725805 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:23.726103 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:23.726106 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:23.726108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:23.726291 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:23.726707 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:23.726714 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:23.726971 (+ 257us) client_negotiation.cc:769] Sending connection context
0929 10:32:23.727166 (+ 195us) client_negotiation.cc:240] Negotiation successful
0929 10:32:23.727511 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.2@KRBTEST.COM" created.
I20260929 10:32:23.729805 7775 heartbeater.cc:347] Connected to a master server at 127.20.92.62:43611
I20260929 10:32:23.730088 7775 heartbeater.cc:464] Registering TS with master...
I20260929 10:32:23.730679 7775 heartbeater.cc:511] Master 127.20.92.62:43611 requested a full tablet report, sending...
I20260929 10:32:23.732708 7584 ts_manager.cc:194] Registered new tserver with Master: ba260a5cf7094549a6172b23aafb01b8 (127.20.92.1:39017)
I20260929 10:32:23.734086 7584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.1@KRBTEST.COM'} at 127.20.92.1:44499
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
Entry for principal kudu/127.20.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab.
I20260929 10:32:23.749619 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:0
--local_ip_for_outbound_sockets=127.20.92.2
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:43611
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:40833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:32:23.860903 7780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:23.861160 7780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:23.861204 7780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:23.865948 7780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:23.866021 7780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:23.866111 7780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:32:23.871503 7780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:40833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.2.keytab
--principal=kudu/127.20.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:43611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:23.872711 7780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:23.874055 7780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:32:23.883821 7780 server_base.cc:1034] running on GCE node
W20260929 10:32:23.883780 7788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:23.883785 7786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:23.884085 7785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:23.884598 7780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:23.885222 7780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:23.886410 7780 hybrid_clock.cc:648] HybridClock initialized: now 1790677943886397 us; error 23 us; skew 500 ppm
Sep 29 10:32:23 dist-test-slave-9cj5 krb5kdc[7538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677943, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:23.889003 7780 init.cc:377] Logged in from keytab as kudu/127.20.92.2@KRBTEST.COM (short username kudu)
I20260929 10:32:23.889984 7780 webserver.cc:460] Webserver started at http://127.20.92.2:36521/ using document root <none> and password file <none>
I20260929 10:32:23.890545 7780 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:23.890640 7780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:23.890873 7780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:23.892872 7780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-1/data/instance:
uuid: "6abbd8b266c34b9ca53371d28ce5e8a5"
format_stamp: "Formatted at 2026-09-29 10:32:23 on dist-test-slave-9cj5"
server_key: "0eac259137847e246e83cc6b1a637e31"
server_key_iv: "1bd765bbbcc5e8c6543dd3fd014d0f8c"
server_key_version: "encryptionkey@0"
I20260929 10:32:23.893425 7780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance:
uuid: "6abbd8b266c34b9ca53371d28ce5e8a5"
format_stamp: "Formatted at 2026-09-29 10:32:23 on dist-test-slave-9cj5"
server_key: "0eac259137847e246e83cc6b1a637e31"
server_key_iv: "1bd765bbbcc5e8c6543dd3fd014d0f8c"
server_key_version: "encryptionkey@0"
I20260929 10:32:23.897112 7780 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260929 10:32:23.899677 7795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:23.901013 7780 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 10:32:23.901160 7780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "6abbd8b266c34b9ca53371d28ce5e8a5"
format_stamp: "Formatted at 2026-09-29 10:32:23 on dist-test-slave-9cj5"
server_key: "0eac259137847e246e83cc6b1a637e31"
server_key_iv: "1bd765bbbcc5e8c6543dd3fd014d0f8c"
server_key_version: "encryptionkey@0"
I20260929 10:32:23.901283 7780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:23.914042 7780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:23.917021 7780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:23.917225 7780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:23.917881 7780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:32:23.918681 7780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:32:23.918753 7780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:23.918823 7780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:32:23.918895 7780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:23.928691 7780 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:46415
I20260929 10:32:23.928707 7908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:46415 every 8 connection(s)
I20260929 10:32:23.929593 7780 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
I20260929 10:32:23.936089 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 7780
I20260929 10:32:23.936221 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance
I20260929 10:32:23.936502 20848 external_mini_cluster.cc:1428] Setting key 24860fbb1dae540e44a9e6413049541b
Sep 29 10:32:23 dist-test-slave-9cj5 krb5kdc[7538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677943, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.2@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:32:23.944106 7623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:23.931477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:39495 (local address 127.20.92.62:43611)
0929 10:32:23.931622 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:23.931625 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:23.932375 (+ 750us) server_negotiation.cc:407] Connection header received
0929 10:32:23.933779 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:23.933783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:23.933844 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:23.933951 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:23.935100 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:23.935622 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:23.937198 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:23.937419 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:23.937527 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:23.940428 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:23.940431 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:23.940433 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:23.940457 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:23.942025 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:23.942561 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:23.942567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:23.942569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:23.942653 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:23.943001 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:23.943004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:23.943006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:23.943192 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:23.943302 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:23.943796 (+ 494us) server_negotiation.cc:299] Negotiation successful
0929 10:32:23.943921 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 10:32:23.944965 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:23.931796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:43611 (local address 127.20.92.2:39495)
0929 10:32:23.932231 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:23.932272 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:23.933533 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:23.934122 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:23.934129 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:23.934479 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:23.934944 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:23.934957 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:23.935747 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:23.935751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:23.937061 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:23.937071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:23.937586 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:23.937591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:23.937803 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:23.938423 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:32:23.938446 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:23.940250 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:23.942149 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:23.942155 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:23.942170 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:23.942445 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:23.942754 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:23.942757 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:23.942759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:23.942898 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:23.943301 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:23.943308 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:23.943563 (+ 255us) client_negotiation.cc:769] Sending connection context
0929 10:32:23.943755 (+ 192us) client_negotiation.cc:240] Negotiation successful
0929 10:32:23.944079 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.20.92.3@KRBTEST.COM; defaulting to no policy
I20260929 10:32:23.946378 7909 heartbeater.cc:347] Connected to a master server at 127.20.92.62:43611
I20260929 10:32:23.946661 7909 heartbeater.cc:464] Registering TS with master...
I20260929 10:32:23.947249 7909 heartbeater.cc:511] Master 127.20.92.62:43611 requested a full tablet report, sending...
I20260929 10:32:23.948289 7584 ts_manager.cc:194] Registered new tserver with Master: 6abbd8b266c34b9ca53371d28ce5e8a5 (127.20.92.2:46415)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.3@KRBTEST.COM" created.
I20260929 10:32:23.948938 7584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.2@KRBTEST.COM'} at 127.20.92.2:39495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
Entry for principal kudu/127.20.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab.
I20260929 10:32:23.964702 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:0
--local_ip_for_outbound_sockets=127.20.92.3
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:43611
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:40833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:32:24.074775 7914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:24.075057 7914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:24.075133 7914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:24.079263 7914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:24.079344 7914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:24.079430 7914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:32:24.084676 7914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:40833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.3.keytab
--principal=kudu/127.20.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:43611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:24.085815 7914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:24.086839 7914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:24.096344 7922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:24.096345 7919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:24.096366 7920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:24.096541 7914 server_base.cc:1034] running on GCE node
I20260929 10:32:24.097051 7914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:24.097675 7914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:24.098858 7914 hybrid_clock.cc:648] HybridClock initialized: now 1790677944098843 us; error 25 us; skew 500 ppm
Sep 29 10:32:24 dist-test-slave-9cj5 krb5kdc[7538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677944, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:24.101702 7914 init.cc:377] Logged in from keytab as kudu/127.20.92.3@KRBTEST.COM (short username kudu)
I20260929 10:32:24.102591 7914 webserver.cc:460] Webserver started at http://127.20.92.3:42159/ using document root <none> and password file <none>
I20260929 10:32:24.103137 7914 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:24.103197 7914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:24.103430 7914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:24.105593 7914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-2/data/instance:
uuid: "8207cace8fb04f778992a5fcec965749"
format_stamp: "Formatted at 2026-09-29 10:32:24 on dist-test-slave-9cj5"
server_key: "0fd653130f24ab81d070ce9ef9de530a"
server_key_iv: "19f7baad9fd203bdf683493a4112f50f"
server_key_version: "encryptionkey@0"
I20260929 10:32:24.106218 7914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance:
uuid: "8207cace8fb04f778992a5fcec965749"
format_stamp: "Formatted at 2026-09-29 10:32:24 on dist-test-slave-9cj5"
server_key: "0fd653130f24ab81d070ce9ef9de530a"
server_key_iv: "19f7baad9fd203bdf683493a4112f50f"
server_key_version: "encryptionkey@0"
I20260929 10:32:24.109989 7914 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 10:32:24.112632 7929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:24.113920 7914 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:32:24.114038 7914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "8207cace8fb04f778992a5fcec965749"
format_stamp: "Formatted at 2026-09-29 10:32:24 on dist-test-slave-9cj5"
server_key: "0fd653130f24ab81d070ce9ef9de530a"
server_key_iv: "19f7baad9fd203bdf683493a4112f50f"
server_key_version: "encryptionkey@0"
I20260929 10:32:24.114149 7914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:24.133631 7914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:24.136785 7914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:24.137006 7914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:24.137612 7914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:32:24.138450 7914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:32:24.138496 7914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:24.138566 7914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:32:24.138604 7914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:24.147550 7914 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:44617
I20260929 10:32:24.147629 8042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:44617 every 8 connection(s)
I20260929 10:32:24.148459 7914 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:32:24.151379 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 7914
I20260929 10:32:24.151535 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance
I20260929 10:32:24.151820 20848 external_mini_cluster.cc:1428] Setting key 25fc7939250e81abfa5ae4b4d3f47920
Sep 29 10:32:24 dist-test-slave-9cj5 krb5kdc[7538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677944, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.3@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:32:24.162842 7623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:24.150364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:60211 (local address 127.20.92.62:43611)
0929 10:32:24.150513 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:24.150517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:24.151291 (+ 774us) server_negotiation.cc:407] Connection header received
0929 10:32:24.153049 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:24.153052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:24.153121 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:24.153211 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:24.154829 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.155347 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.156468 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.156696 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.156777 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:24.159309 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:24.159312 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:24.159315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:24.159340 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:24.160793 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:24.161314 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:24.161318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:24.161319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:24.161420 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:24.161729 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:24.161731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:24.161733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:24.161972 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:24.162052 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:24.162598 (+ 546us) server_negotiation.cc:299] Negotiation successful
0929 10:32:24.162704 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 10:32:24.163676 8045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:24.150637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:43611 (local address 127.20.92.3:60211)
0929 10:32:24.151124 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:24.151166 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:24.152815 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:24.153530 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:24.153540 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:24.154176 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:24.154704 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:24.154718 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.155465 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.155468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:24.156357 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:24.156366 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.156828 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.156831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:24.156969 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:24.157658 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:32:24.157674 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:24.159158 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:24.160906 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:24.160911 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:24.160924 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:24.161201 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:24.161509 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:24.161512 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:24.161514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:24.161647 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:24.162096 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:24.162102 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:24.162337 (+ 235us) client_negotiation.cc:769] Sending connection context
0929 10:32:24.162511 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 10:32:24.162798 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":115,"threads_started":1}
I20260929 10:32:24.165084 8043 heartbeater.cc:347] Connected to a master server at 127.20.92.62:43611
I20260929 10:32:24.165344 8043 heartbeater.cc:464] Registering TS with master...
I20260929 10:32:24.165936 8043 heartbeater.cc:511] Master 127.20.92.62:43611 requested a full tablet report, sending...
I20260929 10:32:24.167017 7584 ts_manager.cc:194] Registered new tserver with Master: 8207cace8fb04f778992a5fcec965749 (127.20.92.3:44617)
I20260929 10:32:24.167586 7584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.92.3@KRBTEST.COM'} at 127.20.92.3:60211
I20260929 10:32:24.178233 20848 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:32:24.188194 7623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:24.180412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42764 (local address 127.20.92.62:43611)
0929 10:32:24.180586 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:24.180590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:24.180785 (+ 195us) server_negotiation.cc:407] Connection header received
0929 10:32:24.180912 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:24.180915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:24.180959 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:24.181048 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:24.181863 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.182385 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.183223 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.183432 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.183511 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:24.184927 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:24.184930 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:24.184932 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:24.184956 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:24.186571 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:24.187000 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:24.187003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:24.187004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:24.187081 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:24.187375 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:24.187378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:24.187379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:24.187653 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:24.187761 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:24.187964 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 10:32:24.188061 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 10:32:24.191103 7584 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:32:24.194239 7584 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:32:24.208762 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:24.203308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33356 (local address 127.20.92.1:39017)
0929 10:32:24.204041 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:24.204055 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:24.204073 (+ 18us) server_negotiation.cc:407] Connection header received
0929 10:32:24.204133 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:24.204137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:24.204311 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:24.204729 (+ 418us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:32:24.204860 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.205662 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.207412 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.208054 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.208177 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:24.208326 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:24.208502 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 10:32:24.208594 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":130,"threads_started":1}
I20260929 10:32:24.208758 8054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:24.202596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:39017 (local address 127.0.0.1:33356)
0929 10:32:24.203358 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:24.203413 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:24.203540 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:24.204456 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:24.204458 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:24.204474 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:32:24.204583 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:24.204589 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.205810 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.205816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:24.207277 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:24.207288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.208185 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.208188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:24.208290 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:24.208324 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 10:32:24.208468 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 10:32:24.208587 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"mutex_wait_us":142,"thread_start_us":97,"threads_started":1}
I20260929 10:32:24.208797 8055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:24.202502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:46415 (local address 127.0.0.1:46168)
0929 10:32:24.203357 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:24.203413 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:24.203562 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:24.204365 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:24.204370 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:24.204394 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:32:24.204553 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:24.204560 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.206111 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.206115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:24.207244 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:24.207254 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.208392 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.208397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:24.208477 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:24.208505 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 10:32:24.208574 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 10:32:24.208640 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":29,"spinlock_wait_cycles":768,"thread_start_us":72,"threads_started":1}
I20260929 10:32:24.209218 8056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:24.202631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46168 (local address 127.20.92.2:46415)
0929 10:32:24.203848 (+ 1217us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:24.203862 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:24.203886 (+ 24us) server_negotiation.cc:407] Connection header received
0929 10:32:24.203966 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:24.203972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:24.204206 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:24.204358 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:32:24.204672 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.205952 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.207364 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.208094 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.208196 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:24.208328 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:24.208972 (+ 644us) server_negotiation.cc:299] Negotiation successful
0929 10:32:24.209086 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1116,"thread_start_us":109,"threads_started":1}
I20260929 10:32:24.209733 8057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:24.202845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:44617 (local address 127.0.0.1:58940)
0929 10:32:24.203755 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:24.203767 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:24.203843 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:24.204539 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:24.204542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:24.204558 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:32:24.204621 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:24.204630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.206129 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.206132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:24.207566 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:24.207574 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.208759 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.208762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:24.209412 (+ 650us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:24.209436 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 10:32:24.209526 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 10:32:24.209604 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"mutex_wait_us":31,"thread_start_us":80,"threads_started":1}
I20260929 10:32:24.210201 8058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:24.203283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58940 (local address 127.20.92.3:44617)
0929 10:32:24.203783 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:24.203798 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:24.203823 (+ 25us) server_negotiation.cc:407] Connection header received
0929 10:32:24.203993 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:24.203999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:24.204197 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:24.204400 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:32:24.204732 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.205990 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.207687 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.208461 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.208583 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:24.208703 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:24.209966 (+ 1263us) server_negotiation.cc:299] Negotiation successful
0929 10:32:24.210059 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":154,"threads_started":1}
I20260929 10:32:24.211890 7709 tablet_service.cc:1467] Processing CreateTablet for tablet 8bdbd523e5de4d86b0a4d0df1450eccd (DEFAULT_TABLE table=test-table [id=cbcdf3729e2b417ab469eed979076285]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:32:24.212216 7843 tablet_service.cc:1467] Processing CreateTablet for tablet 8bdbd523e5de4d86b0a4d0df1450eccd (DEFAULT_TABLE table=test-table [id=cbcdf3729e2b417ab469eed979076285]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:32:24.212783 7709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bdbd523e5de4d86b0a4d0df1450eccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:24.212776 7977 tablet_service.cc:1467] Processing CreateTablet for tablet 8bdbd523e5de4d86b0a4d0df1450eccd (DEFAULT_TABLE table=test-table [id=cbcdf3729e2b417ab469eed979076285]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:32:24.213195 7843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bdbd523e5de4d86b0a4d0df1450eccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:24.213706 7977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bdbd523e5de4d86b0a4d0df1450eccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:24.219480 8060 tablet_bootstrap.cc:492] T 8bdbd523e5de4d86b0a4d0df1450eccd P ba260a5cf7094549a6172b23aafb01b8: Bootstrap starting.
I20260929 10:32:24.220258 8061 tablet_bootstrap.cc:492] T 8bdbd523e5de4d86b0a4d0df1450eccd P 8207cace8fb04f778992a5fcec965749: Bootstrap starting.
I20260929 10:32:24.221333 8062 tablet_bootstrap.cc:492] T 8bdbd523e5de4d86b0a4d0df1450eccd P 6abbd8b266c34b9ca53371d28ce5e8a5: Bootstrap starting.
I20260929 10:32:24.222090 8060 tablet_bootstrap.cc:654] T 8bdbd523e5de4d86b0a4d0df1450eccd P ba260a5cf7094549a6172b23aafb01b8: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:24.222353 8061 tablet_bootstrap.cc:654] T 8bdbd523e5de4d86b0a4d0df1450eccd P 8207cace8fb04f778992a5fcec965749: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:24.223553 8060 log.cc:824] T 8bdbd523e5de4d86b0a4d0df1450eccd P ba260a5cf7094549a6172b23aafb01b8: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:24.223596 8061 log.cc:824] T 8bdbd523e5de4d86b0a4d0df1450eccd P 8207cace8fb04f778992a5fcec965749: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:24.223825 8062 tablet_bootstrap.cc:654] T 8bdbd523e5de4d86b0a4d0df1450eccd P 6abbd8b266c34b9ca53371d28ce5e8a5: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:24.224613 8062 log.cc:824] T 8bdbd523e5de4d86b0a4d0df1450eccd P 6abbd8b266c34b9ca53371d28ce5e8a5: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:24.225720 8061 tablet_bootstrap.cc:492] T 8bdbd523e5de4d86b0a4d0df1450eccd P 8207cace8fb04f778992a5fcec965749: No bootstrap required, opened a new log
I20260929 10:32:24.225716 8060 tablet_bootstrap.cc:492] T 8bdbd523e5de4d86b0a4d0df1450eccd P ba260a5cf7094549a6172b23aafb01b8: No bootstrap required, opened a new log
I20260929 10:32:24.225940 8061 ts_tablet_manager.cc:1397] T 8bdbd523e5de4d86b0a4d0df1450eccd P 8207cace8fb04f778992a5fcec965749: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 10:32:24.225941 8060 ts_tablet_manager.cc:1397] T 8bdbd523e5de4d86b0a4d0df1450eccd P ba260a5cf7094549a6172b23aafb01b8: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 10:32:24.226560 8062 tablet_bootstrap.cc:492] T 8bdbd523e5de4d86b0a4d0df1450eccd P 6abbd8b266c34b9ca53371d28ce5e8a5: No bootstrap required, opened a new log
I20260929 10:32:24.226732 8062 ts_tablet_manager.cc:1397] T 8bdbd523e5de4d86b0a4d0df1450eccd P 6abbd8b266c34b9ca53371d28ce5e8a5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 10:32:24.229089 8060 raft_consensus.cc:348] T 8bdbd523e5de4d86b0a4d0df1450eccd P ba260a5cf7094549a6172b23aafb01b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6abbd8b266c34b9ca53371d28ce5e8a5" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 46415 } } peers { permanent_uuid: "ba260a5cf7094549a6172b23aafb01b8" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39017 } } peers { permanent_uuid: "8207cace8fb04f778992a5fcec965749" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 44617 } }
I20260929 10:32:24.229336 8060 raft_consensus.cc:374] T 8bdbd523e5de4d86b0a4d0df1450eccd P ba260a5cf7094549a6172b23aafb01b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:24.229387 8060 raft_consensus.cc:729] T 8bdbd523e5de4d86b0a4d0df1450eccd P ba260a5cf7094549a6172b23aafb01b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba260a5cf7094549a6172b23aafb01b8, State: Initialized, Role: FOLLOWER
I20260929 10:32:24.229926 8060 consensus_queue.cc:260] T 8bdbd523e5de4d86b0a4d0df1450eccd P ba260a5cf7094549a6172b23aafb01b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6abbd8b266c34b9ca53371d28ce5e8a5" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 46415 } } peers { permanent_uuid: "ba260a5cf7094549a6172b23aafb01b8" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39017 } } peers { permanent_uuid: "8207cace8fb04f778992a5fcec965749" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 44617 } }
I20260929 10:32:24.229897 8061 raft_consensus.cc:348] T 8bdbd523e5de4d86b0a4d0df1450eccd P 8207cace8fb04f778992a5fcec965749 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6abbd8b266c34b9ca53371d28ce5e8a5" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 46415 } } peers { permanent_uuid: "ba260a5cf7094549a6172b23aafb01b8" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39017 } } peers { permanent_uuid: "8207cace8fb04f778992a5fcec965749" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 44617 } }
I20260929 10:32:24.230172 8061 raft_consensus.cc:374] T 8bdbd523e5de4d86b0a4d0df1450eccd P 8207cace8fb04f778992a5fcec965749 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:24.230234 8061 raft_consensus.cc:729] T 8bdbd523e5de4d86b0a4d0df1450eccd P 8207cace8fb04f778992a5fcec965749 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8207cace8fb04f778992a5fcec965749, State: Initialized, Role: FOLLOWER
I20260929 10:32:24.230496 8062 raft_consensus.cc:348] T 8bdbd523e5de4d86b0a4d0df1450eccd P 6abbd8b266c34b9ca53371d28ce5e8a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6abbd8b266c34b9ca53371d28ce5e8a5" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 46415 } } peers { permanent_uuid: "ba260a5cf7094549a6172b23aafb01b8" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39017 } } peers { permanent_uuid: "8207cace8fb04f778992a5fcec965749" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 44617 } }
I20260929 10:32:24.230636 8061 consensus_queue.cc:260] T 8bdbd523e5de4d86b0a4d0df1450eccd P 8207cace8fb04f778992a5fcec965749 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6abbd8b266c34b9ca53371d28ce5e8a5" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 46415 } } peers { permanent_uuid: "ba260a5cf7094549a6172b23aafb01b8" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39017 } } peers { permanent_uuid: "8207cace8fb04f778992a5fcec965749" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 44617 } }
I20260929 10:32:24.230760 8062 raft_consensus.cc:374] T 8bdbd523e5de4d86b0a4d0df1450eccd P 6abbd8b266c34b9ca53371d28ce5e8a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:24.230813 8062 raft_consensus.cc:729] T 8bdbd523e5de4d86b0a4d0df1450eccd P 6abbd8b266c34b9ca53371d28ce5e8a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6abbd8b266c34b9ca53371d28ce5e8a5, State: Initialized, Role: FOLLOWER
I20260929 10:32:24.230926 8060 ts_tablet_manager.cc:1428] T 8bdbd523e5de4d86b0a4d0df1450eccd P ba260a5cf7094549a6172b23aafb01b8: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 10:32:24.231280 7775 heartbeater.cc:503] Master 127.20.92.62:43611 was elected leader, sending a full tablet report...
I20260929 10:32:24.231282 8043 heartbeater.cc:503] Master 127.20.92.62:43611 was elected leader, sending a full tablet report...
I20260929 10:32:24.231552 8061 ts_tablet_manager.cc:1428] T 8bdbd523e5de4d86b0a4d0df1450eccd P 8207cace8fb04f778992a5fcec965749: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 10:32:24.232227 8062 consensus_queue.cc:260] T 8bdbd523e5de4d86b0a4d0df1450eccd P 6abbd8b266c34b9ca53371d28ce5e8a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6abbd8b266c34b9ca53371d28ce5e8a5" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 46415 } } peers { permanent_uuid: "ba260a5cf7094549a6172b23aafb01b8" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39017 } } peers { permanent_uuid: "8207cace8fb04f778992a5fcec965749" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 44617 } }
I20260929 10:32:24.233194 8062 ts_tablet_manager.cc:1428] T 8bdbd523e5de4d86b0a4d0df1450eccd P 6abbd8b266c34b9ca53371d28ce5e8a5: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260929 10:32:24.233534 7909 heartbeater.cc:503] Master 127.20.92.62:43611 was elected leader, sending a full tablet report...
I20260929 10:32:24.253924 8066 raft_consensus.cc:482] T 8bdbd523e5de4d86b0a4d0df1450eccd P ba260a5cf7094549a6172b23aafb01b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:32:24.254151 8066 raft_consensus.cc:504] T 8bdbd523e5de4d86b0a4d0df1450eccd P ba260a5cf7094549a6172b23aafb01b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6abbd8b266c34b9ca53371d28ce5e8a5" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 46415 } } peers { permanent_uuid: "ba260a5cf7094549a6172b23aafb01b8" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39017 } } peers { permanent_uuid: "8207cace8fb04f778992a5fcec965749" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 44617 } }
I20260929 10:32:24.255338 8066 leader_election.cc:288] T 8bdbd523e5de4d86b0a4d0df1450eccd P ba260a5cf7094549a6172b23aafb01b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6abbd8b266c34b9ca53371d28ce5e8a5 (127.20.92.2:46415), 8207cace8fb04f778992a5fcec965749 (127.20.92.3:44617)
I20260929 10:32:24.260979 8056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:24.255671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:47709 (local address 127.20.92.2:46415)
0929 10:32:24.255835 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:24.255840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:24.255901 (+ 61us) server_negotiation.cc:407] Connection header received
0929 10:32:24.256196 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:24.256201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:24.256263 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:24.256355 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:32:24.256753 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.257628 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.259536 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.260178 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.260295 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:24.260394 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:24.260760 (+ 366us) server_negotiation.cc:299] Negotiation successful
0929 10:32:24.260847 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 10:32:24.260979 8058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:24.255827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:60631 (local address 127.20.92.3:44617)
0929 10:32:24.255944 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:24.255948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:24.256187 (+ 239us) server_negotiation.cc:407] Connection header received
0929 10:32:24.256247 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:24.256251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:24.256312 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:24.256390 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:32:24.256753 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.257677 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.259549 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.260176 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.260295 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:24.260376 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:24.260762 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 10:32:24.260845 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 10:32:24.261008 8070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:24.255619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:44617 (local address 127.20.92.1:60631)
0929 10:32:24.256006 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:24.256020 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:24.256111 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:24.256426 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:24.256429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:24.256506 (+ 77us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:32:24.256616 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:24.256621 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.257805 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.257807 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:24.259380 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:24.259395 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.260336 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.260339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:24.260480 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:24.260652 (+ 172us) client_negotiation.cc:769] Sending connection context
0929 10:32:24.260720 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 10:32:24.260821 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":262,"spinlock_wait_cycles":5376,"thread_start_us":117,"threads_started":1}
I20260929 10:32:24.261015 8069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:24.255501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:46415 (local address 127.20.92.1:47709)
0929 10:32:24.255805 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:24.255821 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:24.255984 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:24.256426 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:24.256429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:24.256452 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:32:24.256603 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:24.256611 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.257804 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.257807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:24.259377 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:24.259389 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.260335 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.260339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:24.260484 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:24.260595 (+ 111us) client_negotiation.cc:769] Sending connection context
0929 10:32:24.260691 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 10:32:24.260821 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":198,"thread_start_us":112,"threads_started":1}
I20260929 10:32:24.261785 7997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bdbd523e5de4d86b0a4d0df1450eccd" candidate_uuid: "ba260a5cf7094549a6172b23aafb01b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8207cace8fb04f778992a5fcec965749" is_pre_election: true
I20260929 10:32:24.262144 7997 raft_consensus.cc:2454] T 8bdbd523e5de4d86b0a4d0df1450eccd P 8207cace8fb04f778992a5fcec965749 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba260a5cf7094549a6172b23aafb01b8 in term 0.
I20260929 10:32:24.262168 7863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bdbd523e5de4d86b0a4d0df1450eccd" candidate_uuid: "ba260a5cf7094549a6172b23aafb01b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6abbd8b266c34b9ca53371d28ce5e8a5" is_pre_election: true
I20260929 10:32:24.262428 7863 raft_consensus.cc:2454] T 8bdbd523e5de4d86b0a4d0df1450eccd P 6abbd8b266c34b9ca53371d28ce5e8a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba260a5cf7094549a6172b23aafb01b8 in term 0.
I20260929 10:32:24.262717 7662 leader_election.cc:302] T 8bdbd523e5de4d86b0a4d0df1450eccd P ba260a5cf7094549a6172b23aafb01b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8207cace8fb04f778992a5fcec965749, ba260a5cf7094549a6172b23aafb01b8; no voters:
I20260929 10:32:24.263022 8066 raft_consensus.cc:2781] T 8bdbd523e5de4d86b0a4d0df1450eccd P ba260a5cf7094549a6172b23aafb01b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:32:24.263103 8066 raft_consensus.cc:482] T 8bdbd523e5de4d86b0a4d0df1450eccd P ba260a5cf7094549a6172b23aafb01b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:32:24.263134 8066 raft_consensus.cc:3037] T 8bdbd523e5de4d86b0a4d0df1450eccd P ba260a5cf7094549a6172b23aafb01b8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:24.264222 8066 raft_consensus.cc:504] T 8bdbd523e5de4d86b0a4d0df1450eccd P ba260a5cf7094549a6172b23aafb01b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6abbd8b266c34b9ca53371d28ce5e8a5" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 46415 } } peers { permanent_uuid: "ba260a5cf7094549a6172b23aafb01b8" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39017 } } peers { permanent_uuid: "8207cace8fb04f778992a5fcec965749" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 44617 } }
I20260929 10:32:24.264717 8066 leader_election.cc:288] T 8bdbd523e5de4d86b0a4d0df1450eccd P ba260a5cf7094549a6172b23aafb01b8 [CANDIDATE]: Term 1 election: Requested vote from peers 6abbd8b266c34b9ca53371d28ce5e8a5 (127.20.92.2:46415), 8207cace8fb04f778992a5fcec965749 (127.20.92.3:44617)
I20260929 10:32:24.265110 7863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bdbd523e5de4d86b0a4d0df1450eccd" candidate_uuid: "ba260a5cf7094549a6172b23aafb01b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6abbd8b266c34b9ca53371d28ce5e8a5"
I20260929 10:32:24.265244 7863 raft_consensus.cc:3037] T 8bdbd523e5de4d86b0a4d0df1450eccd P 6abbd8b266c34b9ca53371d28ce5e8a5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:24.265332 7997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bdbd523e5de4d86b0a4d0df1450eccd" candidate_uuid: "ba260a5cf7094549a6172b23aafb01b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8207cace8fb04f778992a5fcec965749"
I20260929 10:32:24.265453 7997 raft_consensus.cc:3037] T 8bdbd523e5de4d86b0a4d0df1450eccd P 8207cace8fb04f778992a5fcec965749 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:24.266635 7863 raft_consensus.cc:2454] T 8bdbd523e5de4d86b0a4d0df1450eccd P 6abbd8b266c34b9ca53371d28ce5e8a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba260a5cf7094549a6172b23aafb01b8 in term 1.
I20260929 10:32:24.266753 7997 raft_consensus.cc:2454] T 8bdbd523e5de4d86b0a4d0df1450eccd P 8207cace8fb04f778992a5fcec965749 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba260a5cf7094549a6172b23aafb01b8 in term 1.
I20260929 10:32:24.267015 7665 leader_election.cc:302] T 8bdbd523e5de4d86b0a4d0df1450eccd P ba260a5cf7094549a6172b23aafb01b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6abbd8b266c34b9ca53371d28ce5e8a5, ba260a5cf7094549a6172b23aafb01b8; no voters:
I20260929 10:32:24.267220 8066 raft_consensus.cc:2781] T 8bdbd523e5de4d86b0a4d0df1450eccd P ba260a5cf7094549a6172b23aafb01b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:32:24.267431 8066 raft_consensus.cc:686] T 8bdbd523e5de4d86b0a4d0df1450eccd P ba260a5cf7094549a6172b23aafb01b8 [term 1 LEADER]: Becoming Leader. State: Replica: ba260a5cf7094549a6172b23aafb01b8, State: Running, Role: LEADER
I20260929 10:32:24.267715 8066 consensus_queue.cc:237] T 8bdbd523e5de4d86b0a4d0df1450eccd P ba260a5cf7094549a6172b23aafb01b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6abbd8b266c34b9ca53371d28ce5e8a5" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 46415 } } peers { permanent_uuid: "ba260a5cf7094549a6172b23aafb01b8" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39017 } } peers { permanent_uuid: "8207cace8fb04f778992a5fcec965749" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 44617 } }
I20260929 10:32:24.270959 7581 catalog_manager.cc:5495] T 8bdbd523e5de4d86b0a4d0df1450eccd P ba260a5cf7094549a6172b23aafb01b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba260a5cf7094549a6172b23aafb01b8 (127.20.92.1). New cstate: current_term: 1 leader_uuid: "ba260a5cf7094549a6172b23aafb01b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6abbd8b266c34b9ca53371d28ce5e8a5" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 46415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba260a5cf7094549a6172b23aafb01b8" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 39017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8207cace8fb04f778992a5fcec965749" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 44617 } health_report { overall_health: UNKNOWN } } }
I20260929 10:32:24.289265 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:24.285578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33362 (local address 127.20.92.1:39017)
0929 10:32:24.285698 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:24.285702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:24.285735 (+ 33us) server_negotiation.cc:407] Connection header received
0929 10:32:24.286066 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:24.286069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:24.286131 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:24.286257 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:32:24.286513 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.287047 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.287897 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.288112 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.288197 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:24.288452 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:32:24.288929 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:32:24.289019 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:24.289105 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 10:32:24.289156 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 10:32:24.295487 7997 raft_consensus.cc:1261] T 8bdbd523e5de4d86b0a4d0df1450eccd P 8207cace8fb04f778992a5fcec965749 [term 1 FOLLOWER]: Refusing update from remote peer ba260a5cf7094549a6172b23aafb01b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:32:24.295504 7863 raft_consensus.cc:1261] T 8bdbd523e5de4d86b0a4d0df1450eccd P 6abbd8b266c34b9ca53371d28ce5e8a5 [term 1 FOLLOWER]: Refusing update from remote peer ba260a5cf7094549a6172b23aafb01b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:32:24.296166 8066 consensus_queue.cc:1035] T 8bdbd523e5de4d86b0a4d0df1450eccd P ba260a5cf7094549a6172b23aafb01b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8207cace8fb04f778992a5fcec965749" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 44617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:32:24.296387 8071 consensus_queue.cc:1035] T 8bdbd523e5de4d86b0a4d0df1450eccd P ba260a5cf7094549a6172b23aafb01b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6abbd8b266c34b9ca53371d28ce5e8a5" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 46415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:32:24.303992 8074 mvcc.cc:205] Tried to move back new op lower bound from 7334616859827298304 to 7334616859722821632. Current Snapshot: MvccSnapshot[applied={T|T < 7334616859827298304}]
I20260929 10:32:24.304508 8076 mvcc.cc:205] Tried to move back new op lower bound from 7334616859827298304 to 7334616859722821632. Current Snapshot: MvccSnapshot[applied={T|T < 7334616859827298304}]
I20260929 10:32:24.305559 8075 mvcc.cc:205] Tried to move back new op lower bound from 7334616859827298304 to 7334616859722821632. Current Snapshot: MvccSnapshot[applied={T|T < 7334616859827298304}]
I20260929 10:32:24.312424 7581 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:32:24.312611 7581 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:32:24.315269 7581 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e15c6948556d4d80a30a9d03af28f299: Sending DeleteTablet for 3 replicas of tablet 8bdbd523e5de4d86b0a4d0df1450eccd
I20260929 10:32:24.316125 7977 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bdbd523e5de4d86b0a4d0df1450eccd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:32:24 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:58940
I20260929 10:32:24.316125 7843 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bdbd523e5de4d86b0a4d0df1450eccd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:32:24 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:46168
I20260929 10:32:24.316795 8083 tablet_replica.cc:330] T 8bdbd523e5de4d86b0a4d0df1450eccd P 6abbd8b266c34b9ca53371d28ce5e8a5: stopping tablet replica
I20260929 10:32:24.316772 7709 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bdbd523e5de4d86b0a4d0df1450eccd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:32:24 UTC) from {username='kudu', principal='kudu/127.20.92.62@KRBTEST.COM'} at 127.0.0.1:33356
I20260929 10:32:24.316881 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 7646
I20260929 10:32:24.317185 8083 raft_consensus.cc:2229] T 8bdbd523e5de4d86b0a4d0df1450eccd P 6abbd8b266c34b9ca53371d28ce5e8a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:32:24.317651 8083 raft_consensus.cc:2258] T 8bdbd523e5de4d86b0a4d0df1450eccd P 6abbd8b266c34b9ca53371d28ce5e8a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:32:24.318650 8084 tablet_replica.cc:330] T 8bdbd523e5de4d86b0a4d0df1450eccd P 8207cace8fb04f778992a5fcec965749: stopping tablet replica
I20260929 10:32:24.318950 8084 raft_consensus.cc:2229] T 8bdbd523e5de4d86b0a4d0df1450eccd P 8207cace8fb04f778992a5fcec965749 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:32:24.319226 8084 raft_consensus.cc:2258] T 8bdbd523e5de4d86b0a4d0df1450eccd P 8207cace8fb04f778992a5fcec965749 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:32:24.319305 8083 ts_tablet_manager.cc:1905] T 8bdbd523e5de4d86b0a4d0df1450eccd P 6abbd8b266c34b9ca53371d28ce5e8a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:32:24.322695 8083 ts_tablet_manager.cc:1918] T 8bdbd523e5de4d86b0a4d0df1450eccd P 6abbd8b266c34b9ca53371d28ce5e8a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:32:24.322803 8083 log.cc:1196] T 8bdbd523e5de4d86b0a4d0df1450eccd P 6abbd8b266c34b9ca53371d28ce5e8a5: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-1/wal/wals/8bdbd523e5de4d86b0a4d0df1450eccd
I20260929 10:32:24.322862 8084 ts_tablet_manager.cc:1905] T 8bdbd523e5de4d86b0a4d0df1450eccd P 8207cace8fb04f778992a5fcec965749: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:32:24.323148 8083 ts_tablet_manager.cc:1939] T 8bdbd523e5de4d86b0a4d0df1450eccd P 6abbd8b266c34b9ca53371d28ce5e8a5: Deleting consensus metadata
I20260929 10:32:24.324535 7570 catalog_manager.cc:4841] TS 6abbd8b266c34b9ca53371d28ce5e8a5 (127.20.92.2:46415): tablet 8bdbd523e5de4d86b0a4d0df1450eccd (table test-table [id=cbcdf3729e2b417ab469eed979076285]) successfully deleted
I20260929 10:32:24.325407 8084 ts_tablet_manager.cc:1918] T 8bdbd523e5de4d86b0a4d0df1450eccd P 8207cace8fb04f778992a5fcec965749: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:32:24.325526 8084 log.cc:1196] T 8bdbd523e5de4d86b0a4d0df1450eccd P 8207cace8fb04f778992a5fcec965749: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677679287822-20848-0/minicluster-data/ts-2/wal/wals/8bdbd523e5de4d86b0a4d0df1450eccd
W20260929 10:32:24.325730 7568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.92.1:39017 (error 108)
I20260929 10:32:24.325879 8084 ts_tablet_manager.cc:1939] T 8bdbd523e5de4d86b0a4d0df1450eccd P 8207cace8fb04f778992a5fcec965749: Deleting consensus metadata
I20260929 10:32:24.326761 8057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:24.326455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:39017 (local address 127.0.0.1:33376)
0929 10:32:24.326586 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:24.326660 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.1:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 10:32:24.326874 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 7780
I20260929 10:32:24.326871 7567 catalog_manager.cc:4841] TS 8207cace8fb04f778992a5fcec965749 (127.20.92.3:44617): tablet 8bdbd523e5de4d86b0a4d0df1450eccd (table test-table [id=cbcdf3729e2b417ab469eed979076285]) successfully deleted
W20260929 10:32:24.326897 7568 catalog_manager.cc:4584] TS ba260a5cf7094549a6172b23aafb01b8 (127.20.92.1:39017): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8bdbd523e5de4d86b0a4d0df1450eccd: Network error: Client connection negotiation failed: client connection to 127.20.92.1:39017: connect: Connection refused (error 111)
I20260929 10:32:24.332564 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 7914
I20260929 10:32:24.337836 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 7551
2026-09-29T10:32:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2286 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T10:32:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:32:24Z Disabled control of system clock
I20260929 10:32:24.369788 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:46525
--webserver_interface=127.20.92.62
--webserver_port=0
--builtin_ntp_servers=127.20.92.20:33935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:46525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 10:32:24.482074 8089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:24.482345 8089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:24.482383 8089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:24.486476 8089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:32:24.486539 8089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:24.486555 8089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:24.486572 8089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:32:24.486588 8089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:32:24.491433 8089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:33935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:46525
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:46525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:24.492581 8089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:24.493993 8089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:24.502372 8094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:24.502442 8095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:24.502377 8097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:24.502617 8089 server_base.cc:1034] running on GCE node
I20260929 10:32:24.503274 8089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:24.504431 8089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:24.505676 8089 hybrid_clock.cc:648] HybridClock initialized: now 1790677944505668 us; error 33 us; skew 500 ppm
I20260929 10:32:24.507609 8089 webserver.cc:460] Webserver started at http://127.20.92.62:33681/ using document root <none> and password file <none>
I20260929 10:32:24.508201 8089 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:24.508296 8089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:24.508531 8089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:24.510690 8089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "6066a544b9864cd6803e39fc3adc6029"
format_stamp: "Formatted at 2026-09-29 10:32:24 on dist-test-slave-9cj5"
server_key: "5b67d676f6a8ff39fd4b140454f1f98b"
server_key_iv: "1263dc7a69556ea595cdc2e487ac4023"
server_key_version: "encryptionkey@0"
I20260929 10:32:24.511267 8089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "6066a544b9864cd6803e39fc3adc6029"
format_stamp: "Formatted at 2026-09-29 10:32:24 on dist-test-slave-9cj5"
server_key: "5b67d676f6a8ff39fd4b140454f1f98b"
server_key_iv: "1263dc7a69556ea595cdc2e487ac4023"
server_key_version: "encryptionkey@0"
I20260929 10:32:24.514919 8089 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:32:24.517352 8103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:24.518736 8089 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:32:24.518873 8089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "6066a544b9864cd6803e39fc3adc6029"
format_stamp: "Formatted at 2026-09-29 10:32:24 on dist-test-slave-9cj5"
server_key: "5b67d676f6a8ff39fd4b140454f1f98b"
server_key_iv: "1263dc7a69556ea595cdc2e487ac4023"
server_key_version: "encryptionkey@0"
I20260929 10:32:24.518985 8089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:24.539119 8089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:24.539918 8089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:24.540126 8089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:24.547204 8089 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:46525
I20260929 10:32:24.547207 8155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:46525 every 8 connection(s)
I20260929 10:32:24.548213 8089 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:32:24.551383 8156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:24.556201 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 8089
I20260929 10:32:24.556329 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:32:24.556627 20848 external_mini_cluster.cc:1428] Setting key 714dfc5cdc82d513d7613e2e7edbd3a1
I20260929 10:32:24.557085 8156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6066a544b9864cd6803e39fc3adc6029: Bootstrap starting.
I20260929 10:32:24.559739 8156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6066a544b9864cd6803e39fc3adc6029: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:24.560617 8156 log.cc:824] T 00000000000000000000000000000000 P 6066a544b9864cd6803e39fc3adc6029: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:24.562557 8156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6066a544b9864cd6803e39fc3adc6029: No bootstrap required, opened a new log
I20260929 10:32:24.566306 8156 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6066a544b9864cd6803e39fc3adc6029 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6066a544b9864cd6803e39fc3adc6029" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 46525 } }
I20260929 10:32:24.566459 8159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:24.558119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40706 (local address 127.20.92.62:46525)
0929 10:32:24.558805 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:24.558819 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:24.558855 (+ 36us) server_negotiation.cc:407] Connection header received
0929 10:32:24.559771 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:24.559800 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:24.560333 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:24.560723 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:24.561161 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.562187 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.563101 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.563563 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.563745 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:24.564710 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:24.564715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:24.564728 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:24.565118 (+ 390us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:24.565175 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:24.565195 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:24.565306 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:24.565454 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 10:32:24.565698 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":132,"threads_started":1}
I20260929 10:32:24.566556 8156 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6066a544b9864cd6803e39fc3adc6029 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:24.566640 8156 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6066a544b9864cd6803e39fc3adc6029 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6066a544b9864cd6803e39fc3adc6029, State: Initialized, Role: FOLLOWER
I20260929 10:32:24.567094 8156 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6066a544b9864cd6803e39fc3adc6029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6066a544b9864cd6803e39fc3adc6029" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 46525 } }
I20260929 10:32:24.567268 8156 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6066a544b9864cd6803e39fc3adc6029 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:32:24.567322 8156 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6066a544b9864cd6803e39fc3adc6029 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:32:24.567401 8156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6066a544b9864cd6803e39fc3adc6029 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:24.568531 8156 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6066a544b9864cd6803e39fc3adc6029 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6066a544b9864cd6803e39fc3adc6029" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 46525 } }
I20260929 10:32:24.568951 8156 leader_election.cc:302] T 00000000000000000000000000000000 P 6066a544b9864cd6803e39fc3adc6029 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6066a544b9864cd6803e39fc3adc6029; no voters:
I20260929 10:32:24.569264 8156 leader_election.cc:288] T 00000000000000000000000000000000 P 6066a544b9864cd6803e39fc3adc6029 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:32:24.569409 8161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6066a544b9864cd6803e39fc3adc6029 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:32:24.569670 8161 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6066a544b9864cd6803e39fc3adc6029 [term 1 LEADER]: Becoming Leader. State: Replica: 6066a544b9864cd6803e39fc3adc6029, State: Running, Role: LEADER
I20260929 10:32:24.569998 8161 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6066a544b9864cd6803e39fc3adc6029 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6066a544b9864cd6803e39fc3adc6029" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 46525 } }
I20260929 10:32:24.570587 8156 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6066a544b9864cd6803e39fc3adc6029 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:32:24.571727 8163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6066a544b9864cd6803e39fc3adc6029 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6066a544b9864cd6803e39fc3adc6029. Latest consensus state: current_term: 1 leader_uuid: "6066a544b9864cd6803e39fc3adc6029" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6066a544b9864cd6803e39fc3adc6029" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 46525 } } }
I20260929 10:32:24.571882 8163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6066a544b9864cd6803e39fc3adc6029 [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:24.572790 8162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6066a544b9864cd6803e39fc3adc6029 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6066a544b9864cd6803e39fc3adc6029" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6066a544b9864cd6803e39fc3adc6029" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 46525 } } }
I20260929 10:32:24.572876 8170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:32:24.572981 8162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6066a544b9864cd6803e39fc3adc6029 [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:24.575676 8170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:32:24.581530 8170 catalog_manager.cc:1347] Generated new cluster ID: e1245f6aefbe4dd7ad5381bde4939f24
I20260929 10:32:24.581617 8170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:32:24.589426 8170 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:32:24.590761 8170 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:32:24.601619 8170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6066a544b9864cd6803e39fc3adc6029: Generated new TSK 0
I20260929 10:32:24.602396 8170 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:32:24.612851 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.1:0
--local_ip_for_outbound_sockets=127.20.92.1
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:46525
--builtin_ntp_servers=127.20.92.20:33935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 10:32:24.729182 8180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:24.729489 8180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:24.729585 8180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:24.733703 8180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:24.733819 8180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:24.733925 8180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.1
I20260929 10:32:24.738767 8180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:33935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.92.1
--webserver_port=0
--tserver_master_addrs=127.20.92.62:46525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.1
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:24.739928 8180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:24.741281 8180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:24.750836 8186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:24.750857 8185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:24.751019 8188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:24.751080 8180 server_base.cc:1034] running on GCE node
I20260929 10:32:24.751585 8180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:24.752202 8180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:24.753371 8180 hybrid_clock.cc:648] HybridClock initialized: now 1790677944753335 us; error 47 us; skew 500 ppm
I20260929 10:32:24.755316 8180 webserver.cc:460] Webserver started at http://127.20.92.1:46803/ using document root <none> and password file <none>
I20260929 10:32:24.755903 8180 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:24.755964 8180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:24.756176 8180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:24.758258 8180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-0/data/instance:
uuid: "720fd5e8188a476eb6156228f1828fa2"
format_stamp: "Formatted at 2026-09-29 10:32:24 on dist-test-slave-9cj5"
server_key: "48eb55fc757ac53262095ef6dff4f723"
server_key_iv: "3f91c116d933c632d8c85af76e37cf67"
server_key_version: "encryptionkey@0"
I20260929 10:32:24.758809 8180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance:
uuid: "720fd5e8188a476eb6156228f1828fa2"
format_stamp: "Formatted at 2026-09-29 10:32:24 on dist-test-slave-9cj5"
server_key: "48eb55fc757ac53262095ef6dff4f723"
server_key_iv: "3f91c116d933c632d8c85af76e37cf67"
server_key_version: "encryptionkey@0"
I20260929 10:32:24.762526 8180 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:32:24.764914 8194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:24.766214 8180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:32:24.766342 8180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-0/wal
uuid: "720fd5e8188a476eb6156228f1828fa2"
format_stamp: "Formatted at 2026-09-29 10:32:24 on dist-test-slave-9cj5"
server_key: "48eb55fc757ac53262095ef6dff4f723"
server_key_iv: "3f91c116d933c632d8c85af76e37cf67"
server_key_version: "encryptionkey@0"
I20260929 10:32:24.766454 8180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:24.781430 8180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:24.782253 8180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:24.782414 8180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:24.782948 8180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:32:24.783893 8180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:32:24.783941 8180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:24.783980 8180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:32:24.783995 8180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:24.792804 8180 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.1:38963
I20260929 10:32:24.792827 8308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.1:38963 every 8 connection(s)
I20260929 10:32:24.793712 8180 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-0/data/info.pb
I20260929 10:32:24.799890 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 8180
I20260929 10:32:24.800014 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-0/wal/instance
I20260929 10:32:24.800355 20848 external_mini_cluster.cc:1428] Setting key 62c17fd65f50ef18482374dcf5dedd09
I20260929 10:32:24.802500 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.2:0
--local_ip_for_outbound_sockets=127.20.92.2
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:46525
--builtin_ntp_servers=127.20.92.20:33935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 10:32:24.803331 8159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:24.795551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.1:33865 (local address 127.20.92.62:46525)
0929 10:32:24.795696 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:24.795699 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:24.796702 (+ 1003us) server_negotiation.cc:407] Connection header received
0929 10:32:24.798079 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:24.798082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:24.798129 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:24.798209 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:24.799075 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.799658 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.801197 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.801421 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.801503 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:24.802760 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:24.802767 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:24.802772 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:24.802789 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:24.802814 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:24.802826 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:24.802877 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:24.803158 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 10:32:24.803216 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 10:32:24.804382 8311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:24.795958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:46525 (local address 127.20.92.1:33865)
0929 10:32:24.796566 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:24.796605 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:24.797866 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:24.798362 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:24.798369 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:24.798452 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:24.798920 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:24.798932 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.799809 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.799813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:24.801053 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:24.801090 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.801597 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.801601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:24.801804 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:24.802427 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:32:24.802429 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:24.802483 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:32:24.802582 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:24.802947 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:24.802959 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:24.802968 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 10:32:24.803096 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 10:32:24.803407 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":108,"threads_started":1}
I20260929 10:32:24.805831 8309 heartbeater.cc:347] Connected to a master server at 127.20.92.62:46525
I20260929 10:32:24.806088 8309 heartbeater.cc:464] Registering TS with master...
I20260929 10:32:24.806646 8309 heartbeater.cc:511] Master 127.20.92.62:46525 requested a full tablet report, sending...
I20260929 10:32:24.808415 8120 ts_manager.cc:194] Registered new tserver with Master: 720fd5e8188a476eb6156228f1828fa2 (127.20.92.1:38963)
I20260929 10:32:24.810246 8120 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.92.1:33865
W20260929 10:32:24.914752 8312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:24.915026 8312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:24.915066 8312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:24.919222 8312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:24.919283 8312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:24.919339 8312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.2
I20260929 10:32:24.924335 8312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:33935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.92.2
--webserver_port=0
--tserver_master_addrs=127.20.92.62:46525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.2
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:24.925364 8312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:24.926770 8312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:24.936210 8320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:24.936285 8317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:24.936362 8312 server_base.cc:1034] running on GCE node
W20260929 10:32:24.936517 8318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:24.936954 8312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:24.937681 8312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:24.938921 8312 hybrid_clock.cc:648] HybridClock initialized: now 1790677944938884 us; error 44 us; skew 500 ppm
I20260929 10:32:24.940866 8312 webserver.cc:460] Webserver started at http://127.20.92.2:38251/ using document root <none> and password file <none>
I20260929 10:32:24.941452 8312 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:24.941552 8312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:24.941847 8312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:24.943914 8312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-1/data/instance:
uuid: "2f483eb4e26e4a5fa35eea3832c4ab4c"
format_stamp: "Formatted at 2026-09-29 10:32:24 on dist-test-slave-9cj5"
server_key: "2216a1c872128361fffe32b95dac932c"
server_key_iv: "aa48cbe5e02f1bf25eaa228166b70bf1"
server_key_version: "encryptionkey@0"
I20260929 10:32:24.944490 8312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance:
uuid: "2f483eb4e26e4a5fa35eea3832c4ab4c"
format_stamp: "Formatted at 2026-09-29 10:32:24 on dist-test-slave-9cj5"
server_key: "2216a1c872128361fffe32b95dac932c"
server_key_iv: "aa48cbe5e02f1bf25eaa228166b70bf1"
server_key_version: "encryptionkey@0"
I20260929 10:32:24.948223 8312 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 10:32:24.950783 8326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:24.951952 8312 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:32:24.952088 8312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-1/wal
uuid: "2f483eb4e26e4a5fa35eea3832c4ab4c"
format_stamp: "Formatted at 2026-09-29 10:32:24 on dist-test-slave-9cj5"
server_key: "2216a1c872128361fffe32b95dac932c"
server_key_iv: "aa48cbe5e02f1bf25eaa228166b70bf1"
server_key_version: "encryptionkey@0"
I20260929 10:32:24.952204 8312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:24.962779 8312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:24.963447 8312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:24.963656 8312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:24.964231 8312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:32:24.965204 8312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:32:24.965284 8312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:24.965363 8312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:32:24.965409 8312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:24.974808 8312 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.2:43997
I20260929 10:32:24.974838 8439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.2:43997 every 8 connection(s)
I20260929 10:32:24.975699 8312 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-1/data/info.pb
I20260929 10:32:24.979722 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 8312
I20260929 10:32:24.979820 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-1/wal/instance
I20260929 10:32:24.980126 20848 external_mini_cluster.cc:1428] Setting key 083c8be25838a94bd5d418937786b906
I20260929 10:32:24.982231 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.92.3:0
--local_ip_for_outbound_sockets=127.20.92.3
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:46525
--builtin_ntp_servers=127.20.92.20:33935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 10:32:24.985236 8159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:24.977417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.2:33241 (local address 127.20.92.62:46525)
0929 10:32:24.977593 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:24.977597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:24.978655 (+ 1058us) server_negotiation.cc:407] Connection header received
0929 10:32:24.980116 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:24.980120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:24.980175 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:24.980268 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:24.981374 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.982047 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.983296 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.983519 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.983629 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:24.984744 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:24.984747 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:24.984750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:24.984770 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:24.984789 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:24.984796 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:24.984872 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:24.985041 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 10:32:24.985100 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 10:32:24.986210 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:24.977877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:46525 (local address 127.20.92.2:33241)
0929 10:32:24.978500 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:24.978539 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:24.979901 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:24.980465 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:24.980534 (+ 69us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:24.980622 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:24.981205 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:24.981223 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.982184 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.982188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:24.983167 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:24.983177 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:24.983665 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:24.983669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:24.983836 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:24.984490 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:32:24.984492 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:24.984530 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:32:24.984590 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:24.984885 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:24.984896 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:24.984903 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 10:32:24.985038 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 10:32:24.985315 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":116,"threads_started":1}
I20260929 10:32:24.987327 8440 heartbeater.cc:347] Connected to a master server at 127.20.92.62:46525
I20260929 10:32:24.987583 8440 heartbeater.cc:464] Registering TS with master...
I20260929 10:32:24.988092 8440 heartbeater.cc:511] Master 127.20.92.62:46525 requested a full tablet report, sending...
I20260929 10:32:24.989142 8120 ts_manager.cc:194] Registered new tserver with Master: 2f483eb4e26e4a5fa35eea3832c4ab4c (127.20.92.2:43997)
I20260929 10:32:24.989704 8120 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.92.2:33241
W20260929 10:32:25.094807 8443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:25.095129 8443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:25.095198 8443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:25.099603 8443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:25.099720 8443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:25.099813 8443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.92.3
I20260929 10:32:25.105011 8443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:33935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.92.3
--webserver_port=0
--tserver_master_addrs=127.20.92.62:46525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.92.3
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:25.106233 8443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:25.107671 8443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:25.117333 8448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:25.117321 8449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:25.117401 8451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:25.117533 8443 server_base.cc:1034] running on GCE node
I20260929 10:32:25.118067 8443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:25.118746 8443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:25.119932 8443 hybrid_clock.cc:648] HybridClock initialized: now 1790677945119904 us; error 38 us; skew 500 ppm
I20260929 10:32:25.121977 8443 webserver.cc:460] Webserver started at http://127.20.92.3:39381/ using document root <none> and password file <none>
I20260929 10:32:25.122561 8443 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:25.122623 8443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:25.122859 8443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:25.124958 8443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-2/data/instance:
uuid: "23d34af784764c95ba11b96be723af6f"
format_stamp: "Formatted at 2026-09-29 10:32:25 on dist-test-slave-9cj5"
server_key: "393feb04cd2487a4ae30c4e766230a6e"
server_key_iv: "5a289887a23e9475917b6292196082a7"
server_key_version: "encryptionkey@0"
I20260929 10:32:25.125475 8443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance:
uuid: "23d34af784764c95ba11b96be723af6f"
format_stamp: "Formatted at 2026-09-29 10:32:25 on dist-test-slave-9cj5"
server_key: "393feb04cd2487a4ae30c4e766230a6e"
server_key_iv: "5a289887a23e9475917b6292196082a7"
server_key_version: "encryptionkey@0"
I20260929 10:32:25.129213 8443 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 10:32:25.131775 8457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:25.133303 8443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:32:25.133414 8443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-2/wal
uuid: "23d34af784764c95ba11b96be723af6f"
format_stamp: "Formatted at 2026-09-29 10:32:25 on dist-test-slave-9cj5"
server_key: "393feb04cd2487a4ae30c4e766230a6e"
server_key_iv: "5a289887a23e9475917b6292196082a7"
server_key_version: "encryptionkey@0"
I20260929 10:32:25.133525 8443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:25.172912 8443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:25.173578 8443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:25.173734 8443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:25.174362 8443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:32:25.175267 8443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:32:25.175315 8443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:25.175355 8443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:32:25.175370 8443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:25.184815 8443 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.3:40235
I20260929 10:32:25.184842 8570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.3:40235 every 8 connection(s)
I20260929 10:32:25.186084 8443 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-2/data/info.pb
I20260929 10:32:25.189157 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 8443
I20260929 10:32:25.189277 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-2/wal/instance
I20260929 10:32:25.189577 20848 external_mini_cluster.cc:1428] Setting key 1315c12ee70ead8e841aeecd4c092044
I20260929 10:32:25.195724 8159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:25.188068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:59949 (local address 127.20.92.62:46525)
0929 10:32:25.188236 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:25.188240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:25.189311 (+ 1071us) server_negotiation.cc:407] Connection header received
0929 10:32:25.191033 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:25.191036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:25.191093 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:25.191190 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:25.192067 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:25.192588 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:25.193711 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:25.193988 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:25.194108 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:25.195229 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:25.195233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:25.195235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:25.195255 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:25.195273 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:25.195281 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:25.195360 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:25.195520 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 10:32:25.195582 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 10:32:25.196592 8573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:25.188485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:46525 (local address 127.20.92.3:59949)
0929 10:32:25.189151 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:25.189190 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:25.190829 (+ 1639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:25.191355 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:25.191361 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:25.191442 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:25.191926 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:25.191938 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:25.192712 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:25.192722 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:25.193586 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:25.193595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:25.194141 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:25.194144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:25.194315 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:25.194955 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:32:25.194957 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:25.194997 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:32:25.195063 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:25.195374 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:25.195382 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:25.195387 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 10:32:25.195529 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 10:32:25.195811 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":124,"threads_started":1}
I20260929 10:32:25.197710 8571 heartbeater.cc:347] Connected to a master server at 127.20.92.62:46525
I20260929 10:32:25.198000 8571 heartbeater.cc:464] Registering TS with master...
I20260929 10:32:25.198549 8571 heartbeater.cc:511] Master 127.20.92.62:46525 requested a full tablet report, sending...
I20260929 10:32:25.199613 8120 ts_manager.cc:194] Registered new tserver with Master: 23d34af784764c95ba11b96be723af6f (127.20.92.3:40235)
I20260929 10:32:25.200138 8120 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.92.3:59949
I20260929 10:32:25.203652 20848 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:32:25.209649 8159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:25.205602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40714 (local address 127.20.92.62:46525)
0929 10:32:25.205788 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:25.205796 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:25.206047 (+ 251us) server_negotiation.cc:407] Connection header received
0929 10:32:25.206127 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:25.206129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:25.206181 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:25.206278 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:25.206511 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:25.207087 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:25.207977 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:25.208213 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:25.208311 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:25.209197 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:25.209200 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:25.209202 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:25.209215 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:25.209233 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:25.209241 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:25.209305 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:25.209410 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 10:32:25.209461 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 10:32:25.212724 8120 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40714:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:32:25.216329 8120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:32:25.231755 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:25.226226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36552 (local address 127.20.92.1:38963)
0929 10:32:25.226542 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:25.226553 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:25.227114 (+ 561us) server_negotiation.cc:407] Connection header received
0929 10:32:25.227193 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:25.227221 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:25.227404 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:25.227837 (+ 433us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:32:25.227971 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:25.228680 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:25.230273 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:25.231037 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:25.231150 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:25.231357 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:25.231465 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 10:32:25.231571 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":114,"threads_started":1}
I20260929 10:32:25.231739 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:25.225652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:38963 (local address 127.0.0.1:36552)
0929 10:32:25.226845 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:25.226859 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:25.226953 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:25.227567 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:25.227573 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:25.227595 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:32:25.227701 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:25.227707 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:25.228798 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:25.228804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:25.230153 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:25.230162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:25.231171 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:25.231174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:25.231276 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:25.231311 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 10:32:25.231433 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 10:32:25.231571 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1133,"mutex_wait_us":115,"thread_start_us":118,"threads_started":1}
I20260929 10:32:25.232061 8584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:25.225595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34462 (local address 127.20.92.3:40235)
0929 10:32:25.226268 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:25.226281 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:25.226847 (+ 566us) server_negotiation.cc:407] Connection header received
0929 10:32:25.227065 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:25.227071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:25.227308 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:25.227479 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:32:25.227784 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:25.228948 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:25.230482 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:25.231213 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:25.231305 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:25.231453 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:25.231810 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 10:32:25.231915 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":196,"threads_started":1}
I20260929 10:32:25.232085 8585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:25.225992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50716 (local address 127.20.92.2:43997)
0929 10:32:25.226379 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:25.226390 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:25.227258 (+ 868us) server_negotiation.cc:407] Connection header received
0929 10:32:25.227333 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:25.227337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:25.227490 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:25.227638 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:32:25.228008 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:25.229120 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:25.230543 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:25.231064 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:25.231195 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:25.231376 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:25.231809 (+ 433us) server_negotiation.cc:299] Negotiation successful
0929 10:32:25.231927 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":145,"threads_started":1}
I20260929 10:32:25.232429 8583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:25.225415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.3:40235 (local address 127.0.0.1:34462)
0929 10:32:25.226706 (+ 1291us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:25.226742 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:25.226897 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:25.227470 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:25.227476 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:25.227496 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:32:25.227639 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:25.227678 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:25.229080 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:25.229086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:25.230352 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:25.230363 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:25.231524 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:25.231528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:25.231633 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:25.231663 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 10:32:25.232221 (+ 558us) client_negotiation.cc:240] Negotiation successful
0929 10:32:25.232306 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1127,"mutex_wait_us":4,"thread_start_us":87,"threads_started":1}
I20260929 10:32:25.232668 8582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:25.226354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:43997 (local address 127.0.0.1:50716)
0929 10:32:25.227134 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:25.227145 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:25.227240 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:25.227779 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:25.227784 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:25.227803 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:32:25.227880 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:25.227886 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:25.229246 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:25.229250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:25.230415 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:25.230425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:25.231438 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:25.231442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:25.231516 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:25.231573 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 10:32:25.231640 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 10:32:25.231695 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":49,"thread_start_us":108,"threads_started":1}
I20260929 10:32:25.235162 8505 tablet_service.cc:1467] Processing CreateTablet for tablet 67aabd39ffa446aba1c2d36b7fcd2525 (DEFAULT_TABLE table=test-table [id=44ddc5b517b04c35905afd01e662ce84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:32:25.235286 8242 tablet_service.cc:1467] Processing CreateTablet for tablet 67aabd39ffa446aba1c2d36b7fcd2525 (DEFAULT_TABLE table=test-table [id=44ddc5b517b04c35905afd01e662ce84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:32:25.235286 8374 tablet_service.cc:1467] Processing CreateTablet for tablet 67aabd39ffa446aba1c2d36b7fcd2525 (DEFAULT_TABLE table=test-table [id=44ddc5b517b04c35905afd01e662ce84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:32:25.236255 8505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67aabd39ffa446aba1c2d36b7fcd2525. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:25.236286 8242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67aabd39ffa446aba1c2d36b7fcd2525. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:25.236286 8374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67aabd39ffa446aba1c2d36b7fcd2525. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:25.242641 8588 tablet_bootstrap.cc:492] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f: Bootstrap starting.
I20260929 10:32:25.243847 8589 tablet_bootstrap.cc:492] T 67aabd39ffa446aba1c2d36b7fcd2525 P 2f483eb4e26e4a5fa35eea3832c4ab4c: Bootstrap starting.
I20260929 10:32:25.245126 8588 tablet_bootstrap.cc:654] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:25.245973 8589 tablet_bootstrap.cc:654] T 67aabd39ffa446aba1c2d36b7fcd2525 P 2f483eb4e26e4a5fa35eea3832c4ab4c: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:25.245993 8588 log.cc:824] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:25.246119 8590 tablet_bootstrap.cc:492] T 67aabd39ffa446aba1c2d36b7fcd2525 P 720fd5e8188a476eb6156228f1828fa2: Bootstrap starting.
I20260929 10:32:25.246773 8589 log.cc:824] T 67aabd39ffa446aba1c2d36b7fcd2525 P 2f483eb4e26e4a5fa35eea3832c4ab4c: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:25.248037 8588 tablet_bootstrap.cc:492] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f: No bootstrap required, opened a new log
I20260929 10:32:25.248243 8588 ts_tablet_manager.cc:1397] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 10:32:25.248273 8590 tablet_bootstrap.cc:654] T 67aabd39ffa446aba1c2d36b7fcd2525 P 720fd5e8188a476eb6156228f1828fa2: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:25.248783 8589 tablet_bootstrap.cc:492] T 67aabd39ffa446aba1c2d36b7fcd2525 P 2f483eb4e26e4a5fa35eea3832c4ab4c: No bootstrap required, opened a new log
I20260929 10:32:25.248965 8589 ts_tablet_manager.cc:1397] T 67aabd39ffa446aba1c2d36b7fcd2525 P 2f483eb4e26e4a5fa35eea3832c4ab4c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 10:32:25.249053 8590 log.cc:824] T 67aabd39ffa446aba1c2d36b7fcd2525 P 720fd5e8188a476eb6156228f1828fa2: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:25.251124 8590 tablet_bootstrap.cc:492] T 67aabd39ffa446aba1c2d36b7fcd2525 P 720fd5e8188a476eb6156228f1828fa2: No bootstrap required, opened a new log
I20260929 10:32:25.251302 8590 ts_tablet_manager.cc:1397] T 67aabd39ffa446aba1c2d36b7fcd2525 P 720fd5e8188a476eb6156228f1828fa2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 10:32:25.252321 8588 raft_consensus.cc:348] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d34af784764c95ba11b96be723af6f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40235 } } peers { permanent_uuid: "720fd5e8188a476eb6156228f1828fa2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 38963 } } peers { permanent_uuid: "2f483eb4e26e4a5fa35eea3832c4ab4c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 43997 } }
I20260929 10:32:25.252719 8588 raft_consensus.cc:374] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:25.252825 8588 raft_consensus.cc:729] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23d34af784764c95ba11b96be723af6f, State: Initialized, Role: FOLLOWER
I20260929 10:32:25.253327 8588 consensus_queue.cc:260] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d34af784764c95ba11b96be723af6f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40235 } } peers { permanent_uuid: "720fd5e8188a476eb6156228f1828fa2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 38963 } } peers { permanent_uuid: "2f483eb4e26e4a5fa35eea3832c4ab4c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 43997 } }
I20260929 10:32:25.253485 8589 raft_consensus.cc:348] T 67aabd39ffa446aba1c2d36b7fcd2525 P 2f483eb4e26e4a5fa35eea3832c4ab4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d34af784764c95ba11b96be723af6f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40235 } } peers { permanent_uuid: "720fd5e8188a476eb6156228f1828fa2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 38963 } } peers { permanent_uuid: "2f483eb4e26e4a5fa35eea3832c4ab4c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 43997 } }
I20260929 10:32:25.253780 8589 raft_consensus.cc:374] T 67aabd39ffa446aba1c2d36b7fcd2525 P 2f483eb4e26e4a5fa35eea3832c4ab4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:25.253845 8589 raft_consensus.cc:729] T 67aabd39ffa446aba1c2d36b7fcd2525 P 2f483eb4e26e4a5fa35eea3832c4ab4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f483eb4e26e4a5fa35eea3832c4ab4c, State: Initialized, Role: FOLLOWER
I20260929 10:32:25.254191 8571 heartbeater.cc:503] Master 127.20.92.62:46525 was elected leader, sending a full tablet report...
I20260929 10:32:25.254330 8589 consensus_queue.cc:260] T 67aabd39ffa446aba1c2d36b7fcd2525 P 2f483eb4e26e4a5fa35eea3832c4ab4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d34af784764c95ba11b96be723af6f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40235 } } peers { permanent_uuid: "720fd5e8188a476eb6156228f1828fa2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 38963 } } peers { permanent_uuid: "2f483eb4e26e4a5fa35eea3832c4ab4c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 43997 } }
I20260929 10:32:25.254416 8588 ts_tablet_manager.cc:1428] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 10:32:25.254978 8440 heartbeater.cc:503] Master 127.20.92.62:46525 was elected leader, sending a full tablet report...
I20260929 10:32:25.255345 8589 ts_tablet_manager.cc:1428] T 67aabd39ffa446aba1c2d36b7fcd2525 P 2f483eb4e26e4a5fa35eea3832c4ab4c: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260929 10:32:25.256412 8590 raft_consensus.cc:348] T 67aabd39ffa446aba1c2d36b7fcd2525 P 720fd5e8188a476eb6156228f1828fa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d34af784764c95ba11b96be723af6f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40235 } } peers { permanent_uuid: "720fd5e8188a476eb6156228f1828fa2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 38963 } } peers { permanent_uuid: "2f483eb4e26e4a5fa35eea3832c4ab4c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 43997 } }
I20260929 10:32:25.256680 8590 raft_consensus.cc:374] T 67aabd39ffa446aba1c2d36b7fcd2525 P 720fd5e8188a476eb6156228f1828fa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:25.256748 8590 raft_consensus.cc:729] T 67aabd39ffa446aba1c2d36b7fcd2525 P 720fd5e8188a476eb6156228f1828fa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 720fd5e8188a476eb6156228f1828fa2, State: Initialized, Role: FOLLOWER
I20260929 10:32:25.257217 8590 consensus_queue.cc:260] T 67aabd39ffa446aba1c2d36b7fcd2525 P 720fd5e8188a476eb6156228f1828fa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d34af784764c95ba11b96be723af6f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40235 } } peers { permanent_uuid: "720fd5e8188a476eb6156228f1828fa2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 38963 } } peers { permanent_uuid: "2f483eb4e26e4a5fa35eea3832c4ab4c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 43997 } }
I20260929 10:32:25.258230 8590 ts_tablet_manager.cc:1428] T 67aabd39ffa446aba1c2d36b7fcd2525 P 720fd5e8188a476eb6156228f1828fa2: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 10:32:25.258666 8309 heartbeater.cc:503] Master 127.20.92.62:46525 was elected leader, sending a full tablet report...
W20260929 10:32:25.294945 8310 tablet.cc:2369] T 67aabd39ffa446aba1c2d36b7fcd2525 P 720fd5e8188a476eb6156228f1828fa2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:32:25.437081 8572 tablet.cc:2369] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:32:25.476795 8441 tablet.cc:2369] T 67aabd39ffa446aba1c2d36b7fcd2525 P 2f483eb4e26e4a5fa35eea3832c4ab4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:32:25.665040 8594 raft_consensus.cc:482] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:32:25.665256 8594 raft_consensus.cc:504] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d34af784764c95ba11b96be723af6f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40235 } } peers { permanent_uuid: "720fd5e8188a476eb6156228f1828fa2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 38963 } } peers { permanent_uuid: "2f483eb4e26e4a5fa35eea3832c4ab4c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 43997 } }
I20260929 10:32:25.666440 8594 leader_election.cc:288] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 720fd5e8188a476eb6156228f1828fa2 (127.20.92.1:38963), 2f483eb4e26e4a5fa35eea3832c4ab4c (127.20.92.2:43997)
I20260929 10:32:25.670625 8573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:25.666796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:38963 (local address 127.20.92.3:52089)
0929 10:32:25.666981 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:25.666998 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:25.667111 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:25.667455 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:25.667459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:25.667482 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:32:25.667621 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:25.667633 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:25.668467 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:25.668471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:25.669359 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:25.669368 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:25.670142 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:25.670146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:25.670253 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:25.670277 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 10:32:25.670354 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 10:32:25.670462 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":2}
I20260929 10:32:25.670624 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:25.667013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:52089 (local address 127.20.92.1:38963)
0929 10:32:25.667149 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:25.667155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:25.667186 (+ 31us) server_negotiation.cc:407] Connection header received
0929 10:32:25.667243 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:25.667247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:25.667308 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:25.667393 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:32:25.667745 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:25.668339 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:25.669486 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:25.670027 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:25.670158 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:25.670243 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:25.670398 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 10:32:25.670471 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 10:32:25.670785 8585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:25.666996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.92.3:40373 (local address 127.20.92.2:43997)
0929 10:32:25.667120 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:25.667124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:25.667433 (+ 309us) server_negotiation.cc:407] Connection header received
0929 10:32:25.667530 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:25.667533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:25.667593 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:25.667695 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:32:25.668022 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:25.668622 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:25.669669 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:25.670165 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:25.670262 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:25.670316 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:25.670593 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 10:32:25.670673 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 10:32:25.670805 8597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:25.666796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.2:43997 (local address 127.20.92.3:40373)
0929 10:32:25.667256 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:25.667269 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:25.667383 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:25.667734 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:25.667736 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:25.667753 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:32:25.667877 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:25.667882 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:25.668738 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:25.668741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:25.669548 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:25.669558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:25.670310 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:25.670313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:25.670467 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:25.670489 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 10:32:25.670565 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 10:32:25.670663 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":169,"thread_start_us":73,"threads_started":1}
I20260929 10:32:25.671337 8263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67aabd39ffa446aba1c2d36b7fcd2525" candidate_uuid: "23d34af784764c95ba11b96be723af6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720fd5e8188a476eb6156228f1828fa2" is_pre_election: true
I20260929 10:32:25.671473 8394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67aabd39ffa446aba1c2d36b7fcd2525" candidate_uuid: "23d34af784764c95ba11b96be723af6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f483eb4e26e4a5fa35eea3832c4ab4c" is_pre_election: true
I20260929 10:32:25.671674 8263 raft_consensus.cc:2454] T 67aabd39ffa446aba1c2d36b7fcd2525 P 720fd5e8188a476eb6156228f1828fa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23d34af784764c95ba11b96be723af6f in term 0.
I20260929 10:32:25.671772 8394 raft_consensus.cc:2454] T 67aabd39ffa446aba1c2d36b7fcd2525 P 2f483eb4e26e4a5fa35eea3832c4ab4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23d34af784764c95ba11b96be723af6f in term 0.
I20260929 10:32:25.672220 8458 leader_election.cc:302] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23d34af784764c95ba11b96be723af6f, 720fd5e8188a476eb6156228f1828fa2; no voters:
I20260929 10:32:25.672488 8594 raft_consensus.cc:2781] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:32:25.672574 8594 raft_consensus.cc:482] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:32:25.672618 8594 raft_consensus.cc:3037] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:25.673707 8594 raft_consensus.cc:504] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d34af784764c95ba11b96be723af6f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40235 } } peers { permanent_uuid: "720fd5e8188a476eb6156228f1828fa2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 38963 } } peers { permanent_uuid: "2f483eb4e26e4a5fa35eea3832c4ab4c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 43997 } }
I20260929 10:32:25.674080 8594 leader_election.cc:288] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f [CANDIDATE]: Term 1 election: Requested vote from peers 720fd5e8188a476eb6156228f1828fa2 (127.20.92.1:38963), 2f483eb4e26e4a5fa35eea3832c4ab4c (127.20.92.2:43997)
I20260929 10:32:25.674427 8394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67aabd39ffa446aba1c2d36b7fcd2525" candidate_uuid: "23d34af784764c95ba11b96be723af6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f483eb4e26e4a5fa35eea3832c4ab4c"
I20260929 10:32:25.674423 8263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67aabd39ffa446aba1c2d36b7fcd2525" candidate_uuid: "23d34af784764c95ba11b96be723af6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720fd5e8188a476eb6156228f1828fa2"
I20260929 10:32:25.674561 8394 raft_consensus.cc:3037] T 67aabd39ffa446aba1c2d36b7fcd2525 P 2f483eb4e26e4a5fa35eea3832c4ab4c [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:25.674561 8263 raft_consensus.cc:3037] T 67aabd39ffa446aba1c2d36b7fcd2525 P 720fd5e8188a476eb6156228f1828fa2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:25.675774 8263 raft_consensus.cc:2454] T 67aabd39ffa446aba1c2d36b7fcd2525 P 720fd5e8188a476eb6156228f1828fa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23d34af784764c95ba11b96be723af6f in term 1.
I20260929 10:32:25.675774 8394 raft_consensus.cc:2454] T 67aabd39ffa446aba1c2d36b7fcd2525 P 2f483eb4e26e4a5fa35eea3832c4ab4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23d34af784764c95ba11b96be723af6f in term 1.
I20260929 10:32:25.676108 8458 leader_election.cc:302] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 23d34af784764c95ba11b96be723af6f, 2f483eb4e26e4a5fa35eea3832c4ab4c, 720fd5e8188a476eb6156228f1828fa2; no voters:
I20260929 10:32:25.676313 8594 raft_consensus.cc:2781] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:32:25.676538 8594 raft_consensus.cc:686] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f [term 1 LEADER]: Becoming Leader. State: Replica: 23d34af784764c95ba11b96be723af6f, State: Running, Role: LEADER
I20260929 10:32:25.676877 8594 consensus_queue.cc:237] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d34af784764c95ba11b96be723af6f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40235 } } peers { permanent_uuid: "720fd5e8188a476eb6156228f1828fa2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 38963 } } peers { permanent_uuid: "2f483eb4e26e4a5fa35eea3832c4ab4c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 43997 } }
I20260929 10:32:25.680531 8118 catalog_manager.cc:5495] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 23d34af784764c95ba11b96be723af6f (127.20.92.3). New cstate: current_term: 1 leader_uuid: "23d34af784764c95ba11b96be723af6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d34af784764c95ba11b96be723af6f" member_type: VOTER last_known_addr { host: "127.20.92.3" port: 40235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "720fd5e8188a476eb6156228f1828fa2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 38963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f483eb4e26e4a5fa35eea3832c4ab4c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 43997 } health_report { overall_health: UNKNOWN } } }
I20260929 10:32:25.799314 8602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:25.795015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34476 (local address 127.20.92.3:40235)
0929 10:32:25.795252 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:25.795256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:25.795313 (+ 57us) server_negotiation.cc:407] Connection header received
0929 10:32:25.795492 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:25.795496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:25.795566 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:25.795686 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:32:25.795941 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:25.796704 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:25.797644 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:25.798021 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:25.798135 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:25.798406 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:32:25.798897 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:32:25.799007 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:25.799113 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 10:32:25.799165 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":71,"threads_started":1}
I20260929 10:32:25.806411 8394 raft_consensus.cc:1261] T 67aabd39ffa446aba1c2d36b7fcd2525 P 2f483eb4e26e4a5fa35eea3832c4ab4c [term 1 FOLLOWER]: Refusing update from remote peer 23d34af784764c95ba11b96be723af6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:32:25.806411 8263 raft_consensus.cc:1261] T 67aabd39ffa446aba1c2d36b7fcd2525 P 720fd5e8188a476eb6156228f1828fa2 [term 1 FOLLOWER]: Refusing update from remote peer 23d34af784764c95ba11b96be723af6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:32:25.807104 8594 consensus_queue.cc:1035] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f483eb4e26e4a5fa35eea3832c4ab4c" member_type: VOTER last_known_addr { host: "127.20.92.2" port: 43997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:32:25.807430 8598 consensus_queue.cc:1035] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "720fd5e8188a476eb6156228f1828fa2" member_type: VOTER last_known_addr { host: "127.20.92.1" port: 38963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:32:25.814181 8603 mvcc.cc:205] Tried to move back new op lower bound from 7334616866014466048 to 7334616865495347200. Current Snapshot: MvccSnapshot[applied={T|T < 7334616866014466048}]
I20260929 10:32:25.816151 8606 mvcc.cc:205] Tried to move back new op lower bound from 7334616866014466048 to 7334616865495347200. Current Snapshot: MvccSnapshot[applied={T|T < 7334616866014466048}]
I20260929 10:32:25.818799 8607 mvcc.cc:205] Tried to move back new op lower bound from 7334616866014466048 to 7334616865495347200. Current Snapshot: MvccSnapshot[applied={T|T < 7334616866014466048}]
I20260929 10:32:25.823966 8120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:32:25.824138 8120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:32:25.826507 8120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6066a544b9864cd6803e39fc3adc6029: Sending DeleteTablet for 3 replicas of tablet 67aabd39ffa446aba1c2d36b7fcd2525
I20260929 10:32:25.827276 8505 tablet_service.cc:1514] Processing DeleteTablet for tablet 67aabd39ffa446aba1c2d36b7fcd2525 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:32:25 UTC) from {username='slave'} at 127.0.0.1:34462
I20260929 10:32:25.827330 8242 tablet_service.cc:1514] Processing DeleteTablet for tablet 67aabd39ffa446aba1c2d36b7fcd2525 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:32:25 UTC) from {username='slave'} at 127.0.0.1:36552
I20260929 10:32:25.827659 8374 tablet_service.cc:1514] Processing DeleteTablet for tablet 67aabd39ffa446aba1c2d36b7fcd2525 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:32:25 UTC) from {username='slave'} at 127.0.0.1:50716
I20260929 10:32:25.828133 8616 tablet_replica.cc:330] T 67aabd39ffa446aba1c2d36b7fcd2525 P 2f483eb4e26e4a5fa35eea3832c4ab4c: stopping tablet replica
I20260929 10:32:25.828487 8615 tablet_replica.cc:330] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f: stopping tablet replica
I20260929 10:32:25.828481 8616 raft_consensus.cc:2229] T 67aabd39ffa446aba1c2d36b7fcd2525 P 2f483eb4e26e4a5fa35eea3832c4ab4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:32:25.828732 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 8180
I20260929 10:32:25.830403 8616 raft_consensus.cc:2258] T 67aabd39ffa446aba1c2d36b7fcd2525 P 2f483eb4e26e4a5fa35eea3832c4ab4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:32:25.831679 8616 ts_tablet_manager.cc:1905] T 67aabd39ffa446aba1c2d36b7fcd2525 P 2f483eb4e26e4a5fa35eea3832c4ab4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:32:25.832819 8615 raft_consensus.cc:2229] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:32:25.833256 8615 raft_consensus.cc:2258] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:32:25.834151 8616 ts_tablet_manager.cc:1918] T 67aabd39ffa446aba1c2d36b7fcd2525 P 2f483eb4e26e4a5fa35eea3832c4ab4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:32:25.834259 8616 log.cc:1196] T 67aabd39ffa446aba1c2d36b7fcd2525 P 2f483eb4e26e4a5fa35eea3832c4ab4c: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-1/wal/wals/67aabd39ffa446aba1c2d36b7fcd2525
I20260929 10:32:25.834548 8616 ts_tablet_manager.cc:1939] T 67aabd39ffa446aba1c2d36b7fcd2525 P 2f483eb4e26e4a5fa35eea3832c4ab4c: Deleting consensus metadata
I20260929 10:32:25.835645 8105 catalog_manager.cc:4841] TS 2f483eb4e26e4a5fa35eea3832c4ab4c (127.20.92.2:43997): tablet 67aabd39ffa446aba1c2d36b7fcd2525 (table test-table [id=44ddc5b517b04c35905afd01e662ce84]) successfully deleted
I20260929 10:32:25.836608 8615 ts_tablet_manager.cc:1905] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 10:32:25.839157 8104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.92.1:38963 (error 108)
I20260929 10:32:25.839273 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 8312
I20260929 10:32:25.841619 8615 ts_tablet_manager.cc:1918] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:32:25.841728 8615 log.cc:1196] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f: Deleting WAL directory at /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677679287822-20848-0/minicluster-data/ts-2/wal/wals/67aabd39ffa446aba1c2d36b7fcd2525
I20260929 10:32:25.842097 8615 ts_tablet_manager.cc:1939] T 67aabd39ffa446aba1c2d36b7fcd2525 P 23d34af784764c95ba11b96be723af6f: Deleting consensus metadata
I20260929 10:32:25.843456 8106 catalog_manager.cc:4841] TS 23d34af784764c95ba11b96be723af6f (127.20.92.3:40235): tablet 67aabd39ffa446aba1c2d36b7fcd2525 (table test-table [id=44ddc5b517b04c35905afd01e662ce84]) successfully deleted
I20260929 10:32:25.845490 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 8443
I20260929 10:32:25.847067 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:25.846591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.1:38963 (local address 127.0.0.1:36568)
0929 10:32:25.846899 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:25.846981 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.1:38963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":70,"threads_started":1}
W20260929 10:32:25.847249 8104 catalog_manager.cc:4584] TS 720fd5e8188a476eb6156228f1828fa2 (127.20.92.1:38963): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 67aabd39ffa446aba1c2d36b7fcd2525: Network error: Client connection negotiation failed: client connection to 127.20.92.1:38963: connect: Connection refused (error 111)
I20260929 10:32:25.851574 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 8089
2026-09-29T10:32:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1511 ms)
[----------] 6 tests from EncryptionPolicyTest (10951 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:32:25 dist-test-slave-9cj5 krb5kdc[8621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:32:25 dist-test-slave-9cj5 krb5kdc[8621](info): set up 2 sockets
Sep 29 10:32:25 dist-test-slave-9cj5 krb5kdc[8621](info): commencing operation
krb5kdc: starting...
W20260929 10:32:26.927076 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:32:27 dist-test-slave-9cj5 krb5kdc[8621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:32:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:32:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:32:27.058192 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38803
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:32:27.167630 8634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:27.167909 8634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:27.167953 8634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:27.172109 8634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:32:27.172187 8634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:27.172209 8634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:27.172225 8634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:32:27.172241 8634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:32:27.177913 8634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38803
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:27.179075 8634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:27.180140 8634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:27.188861 8642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:27.188849 8639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:27.188905 8634 server_base.cc:1034] running on GCE node
W20260929 10:32:27.188849 8640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:27.189606 8634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:27.190768 8634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:27.191944 8634 hybrid_clock.cc:648] HybridClock initialized: now 1790677947191912 us; error 46 us; skew 500 ppm
Sep 29 10:32:27 dist-test-slave-9cj5 krb5kdc[8621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:27.194981 8634 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:32:27.196012 8634 webserver.cc:460] Webserver started at http://127.0.0.1:37123/ using document root <none> and password file <none>
I20260929 10:32:27.196583 8634 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:27.196653 8634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:27.196873 8634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:27.199033 8634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "9351145396bc4485905c73765fb70e02"
format_stamp: "Formatted at 2026-09-29 10:32:27 on dist-test-slave-9cj5"
server_key: "ecca425690cd380d441899b7dce43b63"
server_key_iv: "1ed07e116bf2ef42ec75db610f8178d2"
server_key_version: "encryptionkey@0"
I20260929 10:32:27.199615 8634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "9351145396bc4485905c73765fb70e02"
format_stamp: "Formatted at 2026-09-29 10:32:27 on dist-test-slave-9cj5"
server_key: "ecca425690cd380d441899b7dce43b63"
server_key_iv: "1ed07e116bf2ef42ec75db610f8178d2"
server_key_version: "encryptionkey@0"
I20260929 10:32:27.203735 8634 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:32:27.206333 8649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:27.207778 8634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:32:27.207945 8634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "9351145396bc4485905c73765fb70e02"
format_stamp: "Formatted at 2026-09-29 10:32:27 on dist-test-slave-9cj5"
server_key: "ecca425690cd380d441899b7dce43b63"
server_key_iv: "1ed07e116bf2ef42ec75db610f8178d2"
server_key_version: "encryptionkey@0"
I20260929 10:32:27.208073 8634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:27.221993 8634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:27.225324 8634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:27.225553 8634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:27.233397 8634 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38803
I20260929 10:32:27.233407 8701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38803 every 8 connection(s)
I20260929 10:32:27.234565 8634 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:32:27.234941 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 8634
I20260929 10:32:27.235110 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:32:27.235419 20848 external_mini_cluster.cc:1428] Setting key c6e0687cbae712276e32b39df6ce1149
I20260929 10:32:27.237991 8702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:27.244859 8702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9351145396bc4485905c73765fb70e02: Bootstrap starting.
Sep 29 10:32:27 dist-test-slave-9cj5 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 10:32:27.247957 8702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9351145396bc4485905c73765fb70e02: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:27.248905 8702 log.cc:824] T 00000000000000000000000000000000 P 9351145396bc4485905c73765fb70e02: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:27.251356 8702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9351145396bc4485905c73765fb70e02: No bootstrap required, opened a new log
I20260929 10:32:27.251616 8705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:27.236917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37921 (local address 127.0.0.1:38803)
0929 10:32:27.237390 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:27.237404 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:27.237441 (+ 37us) server_negotiation.cc:407] Connection header received
0929 10:32:27.238253 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:27.238272 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:27.238615 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:27.238957 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:27.239891 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:27.241680 (+ 1789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:27.242578 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:27.243326 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:27.243498 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:27.246088 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:27.246097 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:27.246118 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:27.246164 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:27.248251 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:27.248812 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:27.248818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:27.248824 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:27.248972 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:27.249385 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:27.249389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:27.249391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:27.249897 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:27.250128 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:27.250365 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 10:32:27.250650 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":167,"threads_started":1}
I20260929 10:32:27.254709 20848 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:32:27.255966 8702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9351145396bc4485905c73765fb70e02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9351145396bc4485905c73765fb70e02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38803 } }
I20260929 10:32:27.256206 8702 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9351145396bc4485905c73765fb70e02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:27.256286 8702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9351145396bc4485905c73765fb70e02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9351145396bc4485905c73765fb70e02, State: Initialized, Role: FOLLOWER
I20260929 10:32:27.256807 8702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9351145396bc4485905c73765fb70e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9351145396bc4485905c73765fb70e02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38803 } }
I20260929 10:32:27.256984 8702 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9351145396bc4485905c73765fb70e02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:32:27.257076 8702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9351145396bc4485905c73765fb70e02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:32:27.257192 8702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9351145396bc4485905c73765fb70e02 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:27.258674 8702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9351145396bc4485905c73765fb70e02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9351145396bc4485905c73765fb70e02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38803 } }
I20260929 10:32:27.259049 8702 leader_election.cc:302] T 00000000000000000000000000000000 P 9351145396bc4485905c73765fb70e02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9351145396bc4485905c73765fb70e02; no voters:
I20260929 10:32:27.259416 8702 leader_election.cc:288] T 00000000000000000000000000000000 P 9351145396bc4485905c73765fb70e02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:32:27.259541 8712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9351145396bc4485905c73765fb70e02 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:32:27.259785 8712 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9351145396bc4485905c73765fb70e02 [term 1 LEADER]: Becoming Leader. State: Replica: 9351145396bc4485905c73765fb70e02, State: Running, Role: LEADER
I20260929 10:32:27.260103 8712 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9351145396bc4485905c73765fb70e02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9351145396bc4485905c73765fb70e02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38803 } }
I20260929 10:32:27.260816 8702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9351145396bc4485905c73765fb70e02 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:32:27.261224 8712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9351145396bc4485905c73765fb70e02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9351145396bc4485905c73765fb70e02. Latest consensus state: current_term: 1 leader_uuid: "9351145396bc4485905c73765fb70e02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9351145396bc4485905c73765fb70e02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38803 } } }
I20260929 10:32:27.261348 8712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9351145396bc4485905c73765fb70e02 [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:27.262002 8716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:32:27.262174 8714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9351145396bc4485905c73765fb70e02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9351145396bc4485905c73765fb70e02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9351145396bc4485905c73765fb70e02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38803 } } }
I20260929 10:32:27.262305 8714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9351145396bc4485905c73765fb70e02 [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:27.265175 8716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:32:27.267596 8705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:27.256844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50739 (local address 127.0.0.1:38803)
0929 10:32:27.256979 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:27.256984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:27.257285 (+ 301us) server_negotiation.cc:407] Connection header received
0929 10:32:27.257411 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:27.257413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:27.257453 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:27.257517 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:27.258432 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:27.259038 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:27.260258 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:27.260623 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:27.260706 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:27.262874 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:27.262878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:27.262881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:27.262907 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:27.264743 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:27.265372 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:27.265376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:27.265378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:27.265469 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:27.266014 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:27.266018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:27.266020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:27.266258 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:27.266409 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:27.267307 (+ 898us) server_negotiation.cc:299] Negotiation successful
0929 10:32:27.267413 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 10:32:27.271154 8716 catalog_manager.cc:1347] Generated new cluster ID: e21e96fed3974656abffe64564b49057
I20260929 10:32:27.271227 8716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:32:27.297248 8716 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:32:27.298307 8716 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:32:27.308945 8716 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9351145396bc4485905c73765fb70e02: Generated new TSK 0
I20260929 10:32:27.309689 8716 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:32:27.324394 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 8634
W20260929 10:32:27.329639 8619 connection.cc:535] client connection to 127.0.0.1:38803 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38803): Connection reset by peer (error 104)
2026-09-29T10:32:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1470 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:32:27 dist-test-slave-9cj5 krb5kdc[8733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:32:27 dist-test-slave-9cj5 krb5kdc[8733](info): set up 2 sockets
Sep 29 10:32:27 dist-test-slave-9cj5 krb5kdc[8733](info): commencing operation
krb5kdc: starting...
W20260929 10:32:28.375070 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:32:28 dist-test-slave-9cj5 krb5kdc[8733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:32:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:32:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:32:28.509665 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34763
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:32:28.621380 8746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:28.621655 8746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:28.621698 8746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:28.626613 8746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:32:28.626693 8746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:28.626713 8746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:28.626732 8746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:32:28.626748 8746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:32:28.632225 8746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34763
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:28.633260 8746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:28.634238 8746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:28.643004 8752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:28.642947 8751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:28.642947 8754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:28.643107 8746 server_base.cc:1034] running on GCE node
I20260929 10:32:28.643775 8746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:28.644889 8746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:28.646101 8746 hybrid_clock.cc:648] HybridClock initialized: now 1790677948646071 us; error 41 us; skew 500 ppm
Sep 29 10:32:28 dist-test-slave-9cj5 krb5kdc[8733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:28.649204 8746 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:32:28.650219 8746 webserver.cc:460] Webserver started at http://127.0.0.1:39781/ using document root <none> and password file <none>
I20260929 10:32:28.650777 8746 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:28.650840 8746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:28.651016 8746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:28.653115 8746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "6ff2d418e7524976b36b1726e8842445"
format_stamp: "Formatted at 2026-09-29 10:32:28 on dist-test-slave-9cj5"
server_key: "fc2752c30e57ee979bd1ce04dc10437e"
server_key_iv: "7d08c2e8db146e920d68a66d037a64c2"
server_key_version: "encryptionkey@0"
I20260929 10:32:28.653664 8746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "6ff2d418e7524976b36b1726e8842445"
format_stamp: "Formatted at 2026-09-29 10:32:28 on dist-test-slave-9cj5"
server_key: "fc2752c30e57ee979bd1ce04dc10437e"
server_key_iv: "7d08c2e8db146e920d68a66d037a64c2"
server_key_version: "encryptionkey@0"
I20260929 10:32:28.657663 8746 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 10:32:28.660347 8761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:28.661676 8746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:32:28.661857 8746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "6ff2d418e7524976b36b1726e8842445"
format_stamp: "Formatted at 2026-09-29 10:32:28 on dist-test-slave-9cj5"
server_key: "fc2752c30e57ee979bd1ce04dc10437e"
server_key_iv: "7d08c2e8db146e920d68a66d037a64c2"
server_key_version: "encryptionkey@0"
I20260929 10:32:28.661976 8746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:28.689363 8746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:28.692729 8746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:28.692893 8746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:28.700111 8746 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34763
I20260929 10:32:28.700129 8813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34763 every 8 connection(s)
I20260929 10:32:28.701040 8746 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:32:28.703722 8814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:28.706543 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 8746
I20260929 10:32:28.706727 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:32:28.707038 20848 external_mini_cluster.cc:1428] Setting key d60d78e9247dc4bdb1fbe42ef63a6954
I20260929 10:32:28.709556 8814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ff2d418e7524976b36b1726e8842445: Bootstrap starting.
I20260929 10:32:28.711712 8814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ff2d418e7524976b36b1726e8842445: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:28.712442 8814 log.cc:824] T 00000000000000000000000000000000 P 6ff2d418e7524976b36b1726e8842445: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:28.714507 8814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ff2d418e7524976b36b1726e8842445: No bootstrap required, opened a new log
I20260929 10:32:28.715987 8817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:28.708422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41793 (local address 127.0.0.1:34763)
0929 10:32:28.708861 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:28.708875 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:28.708909 (+ 34us) server_negotiation.cc:407] Connection header received
0929 10:32:28.709497 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:28.709512 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:28.709938 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:28.710268 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:28.710675 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:28.711645 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:28.712572 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:28.713027 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:28.713214 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:28.714287 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:28.714293 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:28.714304 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:28.714770 (+ 466us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:28.714827 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:28.714850 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:28.714995 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:28.715183 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 10:32:28.715316 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":132,"threads_started":1}
I20260929 10:32:28.718163 8814 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ff2d418e7524976b36b1726e8842445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff2d418e7524976b36b1726e8842445" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34763 } }
I20260929 10:32:28.718422 8814 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ff2d418e7524976b36b1726e8842445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:28.718511 8814 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ff2d418e7524976b36b1726e8842445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff2d418e7524976b36b1726e8842445, State: Initialized, Role: FOLLOWER
I20260929 10:32:28.718886 20848 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:32:28.719014 8814 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ff2d418e7524976b36b1726e8842445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff2d418e7524976b36b1726e8842445" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34763 } }
I20260929 10:32:28.719187 8814 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ff2d418e7524976b36b1726e8842445 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:32:28.719250 8814 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ff2d418e7524976b36b1726e8842445 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:32:28.719342 8814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ff2d418e7524976b36b1726e8842445 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:28.720330 8814 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ff2d418e7524976b36b1726e8842445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff2d418e7524976b36b1726e8842445" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34763 } }
I20260929 10:32:28.720665 8814 leader_election.cc:302] T 00000000000000000000000000000000 P 6ff2d418e7524976b36b1726e8842445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ff2d418e7524976b36b1726e8842445; no voters:
I20260929 10:32:28.720957 8814 leader_election.cc:288] T 00000000000000000000000000000000 P 6ff2d418e7524976b36b1726e8842445 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:32:28.721104 8823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ff2d418e7524976b36b1726e8842445 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:32:28.721367 8823 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ff2d418e7524976b36b1726e8842445 [term 1 LEADER]: Becoming Leader. State: Replica: 6ff2d418e7524976b36b1726e8842445, State: Running, Role: LEADER
I20260929 10:32:28.721685 8823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ff2d418e7524976b36b1726e8842445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff2d418e7524976b36b1726e8842445" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34763 } }
I20260929 10:32:28.722436 8814 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ff2d418e7524976b36b1726e8842445 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:32:28.723459 8825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff2d418e7524976b36b1726e8842445 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ff2d418e7524976b36b1726e8842445" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff2d418e7524976b36b1726e8842445" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34763 } } }
I20260929 10:32:28.723639 8825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff2d418e7524976b36b1726e8842445 [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:28.723896 8825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff2d418e7524976b36b1726e8842445 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ff2d418e7524976b36b1726e8842445. Latest consensus state: current_term: 1 leader_uuid: "6ff2d418e7524976b36b1726e8842445" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff2d418e7524976b36b1726e8842445" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34763 } } }
I20260929 10:32:28.724044 8825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff2d418e7524976b36b1726e8842445 [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:28.725090 8833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:32:28.726478 8817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:28.721262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34951 (local address 127.0.0.1:34763)
0929 10:32:28.721375 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:28.721381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:28.721397 (+ 16us) server_negotiation.cc:407] Connection header received
0929 10:32:28.721435 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:28.721437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:28.721481 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:28.721566 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:28.721986 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:28.722824 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:28.724043 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:28.724383 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:28.724466 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:28.725904 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:28.725907 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:28.725909 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:28.725922 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:28.725947 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:28.725958 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:28.726037 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:28.726341 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 10:32:28.726374 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 10:32:28.728878 8833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:32:28.735075 8833 catalog_manager.cc:1347] Generated new cluster ID: d368fcf5ff3a4e78af8158b72fca9fed
I20260929 10:32:28.735198 8833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:32:28.765312 8833 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:32:28.766688 8833 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:32:28.774044 8833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ff2d418e7524976b36b1726e8842445: Generated new TSK 0
I20260929 10:32:28.774892 8833 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:32:28.790139 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 8746
W20260929 10:32:28.795478 8731 connection.cc:535] client connection to 127.0.0.1:34763 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34763): Connection reset by peer (error 104)
2026-09-29T10:32:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1465 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:32:28 dist-test-slave-9cj5 krb5kdc[8845](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:32:28 dist-test-slave-9cj5 krb5kdc[8845](info): set up 2 sockets
Sep 29 10:32:28 dist-test-slave-9cj5 krb5kdc[8845](info): commencing operation
krb5kdc: starting...
W20260929 10:32:29.851379 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:32:29 dist-test-slave-9cj5 krb5kdc[8845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:32:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:32:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:32:29.977438 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40917
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:32:30.092146 8858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:30.092473 8858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:30.092523 8858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:30.097126 8858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:32:30.097210 8858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:30.097232 8858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:30.097252 8858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:32:30.097270 8858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:32:30.103063 8858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40917
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:30.104158 8858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:30.105180 8858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:30.113848 8863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:30.113866 8866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:30.113874 8864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:30.114207 8858 server_base.cc:1034] running on GCE node
I20260929 10:32:30.114804 8858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:30.115960 8858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:30.117153 8858 hybrid_clock.cc:648] HybridClock initialized: now 1790677950117121 us; error 45 us; skew 500 ppm
Sep 29 10:32:30 dist-test-slave-9cj5 krb5kdc[8845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:30.119957 8858 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:32:30.120885 8858 webserver.cc:460] Webserver started at http://127.0.0.1:45705/ using document root <none> and password file <none>
I20260929 10:32:30.121467 8858 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:30.121560 8858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:30.121856 8858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:30.123930 8858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "d924da185d6d4bd7a2b03c55be94261d"
format_stamp: "Formatted at 2026-09-29 10:32:30 on dist-test-slave-9cj5"
server_key: "ec4c2250dae3953bd2c888da1b968328"
server_key_iv: "5eb3acf1cdaa99fcdc6817c102ff1f66"
server_key_version: "encryptionkey@0"
I20260929 10:32:30.124521 8858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "d924da185d6d4bd7a2b03c55be94261d"
format_stamp: "Formatted at 2026-09-29 10:32:30 on dist-test-slave-9cj5"
server_key: "ec4c2250dae3953bd2c888da1b968328"
server_key_iv: "5eb3acf1cdaa99fcdc6817c102ff1f66"
server_key_version: "encryptionkey@0"
I20260929 10:32:30.128412 8858 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 10:32:30.130837 8873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:30.132172 8858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:32:30.132342 8858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "d924da185d6d4bd7a2b03c55be94261d"
format_stamp: "Formatted at 2026-09-29 10:32:30 on dist-test-slave-9cj5"
server_key: "ec4c2250dae3953bd2c888da1b968328"
server_key_iv: "5eb3acf1cdaa99fcdc6817c102ff1f66"
server_key_version: "encryptionkey@0"
I20260929 10:32:30.132472 8858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:30.137480 8858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:30.141003 8858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:30.141238 8858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:30.148707 8858 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40917
I20260929 10:32:30.148725 8925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40917 every 8 connection(s)
I20260929 10:32:30.149657 8858 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:32:30.152473 8926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:30.154043 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 8858
I20260929 10:32:30.154189 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:32:30.154484 20848 external_mini_cluster.cc:1428] Setting key c666087af0c9bf11f8e2a2f031bca902
I20260929 10:32:30.159449 8926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d924da185d6d4bd7a2b03c55be94261d: Bootstrap starting.
I20260929 10:32:30.160368 8929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:30.155913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55763 (local address 127.0.0.1:40917)
0929 10:32:30.156369 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:30.156391 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:30.156437 (+ 46us) server_negotiation.cc:407] Connection header received
0929 10:32:30.157451 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:30.157476 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:30.157867 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:30.158248 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:30.159006 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:30.159013 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:30.159028 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:30.159040 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:30.159102 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:30.159128 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:30.159268 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:30.159335 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 10:32:30.159485 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":111,"threads_started":1}
I20260929 10:32:30.162048 8926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d924da185d6d4bd7a2b03c55be94261d: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:30.162863 8926 log.cc:824] T 00000000000000000000000000000000 P d924da185d6d4bd7a2b03c55be94261d: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:30.163529 20848 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:32:30.165282 8926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d924da185d6d4bd7a2b03c55be94261d: No bootstrap required, opened a new log
I20260929 10:32:30.167804 8929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:30.165867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55473 (local address 127.0.0.1:40917)
0929 10:32:30.165995 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:30.166000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:30.166200 (+ 200us) server_negotiation.cc:407] Connection header received
0929 10:32:30.166294 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:30.166297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:30.166332 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:30.166409 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:30.167332 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:30.167336 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:30.167339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:30.167350 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:30.167377 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:30.167389 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:30.167475 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:30.167643 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 10:32:30.167682 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 10:32:30.169344 8926 raft_consensus.cc:348] T 00000000000000000000000000000000 P d924da185d6d4bd7a2b03c55be94261d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d924da185d6d4bd7a2b03c55be94261d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40917 } }
I20260929 10:32:30.169559 8926 raft_consensus.cc:374] T 00000000000000000000000000000000 P d924da185d6d4bd7a2b03c55be94261d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:30.169618 8926 raft_consensus.cc:729] T 00000000000000000000000000000000 P d924da185d6d4bd7a2b03c55be94261d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d924da185d6d4bd7a2b03c55be94261d, State: Initialized, Role: FOLLOWER
I20260929 10:32:30.170136 8926 consensus_queue.cc:260] T 00000000000000000000000000000000 P d924da185d6d4bd7a2b03c55be94261d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d924da185d6d4bd7a2b03c55be94261d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40917 } }
I20260929 10:32:30.170259 8926 raft_consensus.cc:388] T 00000000000000000000000000000000 P d924da185d6d4bd7a2b03c55be94261d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:32:30.170346 8926 raft_consensus.cc:482] T 00000000000000000000000000000000 P d924da185d6d4bd7a2b03c55be94261d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:32:30.170439 8926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d924da185d6d4bd7a2b03c55be94261d [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:30.171370 8926 raft_consensus.cc:504] T 00000000000000000000000000000000 P d924da185d6d4bd7a2b03c55be94261d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d924da185d6d4bd7a2b03c55be94261d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40917 } }
I20260929 10:32:30.171726 8926 leader_election.cc:302] T 00000000000000000000000000000000 P d924da185d6d4bd7a2b03c55be94261d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d924da185d6d4bd7a2b03c55be94261d; no voters:
I20260929 10:32:30.172029 8926 leader_election.cc:288] T 00000000000000000000000000000000 P d924da185d6d4bd7a2b03c55be94261d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:32:30.172351 8936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d924da185d6d4bd7a2b03c55be94261d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:32:30.172547 8936 raft_consensus.cc:686] T 00000000000000000000000000000000 P d924da185d6d4bd7a2b03c55be94261d [term 1 LEADER]: Becoming Leader. State: Replica: d924da185d6d4bd7a2b03c55be94261d, State: Running, Role: LEADER
I20260929 10:32:30.172899 8936 consensus_queue.cc:237] T 00000000000000000000000000000000 P d924da185d6d4bd7a2b03c55be94261d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d924da185d6d4bd7a2b03c55be94261d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40917 } }
I20260929 10:32:30.173270 8926 sys_catalog.cc:564] T 00000000000000000000000000000000 P d924da185d6d4bd7a2b03c55be94261d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:32:30.174682 8938 sys_catalog.cc:455] T 00000000000000000000000000000000 P d924da185d6d4bd7a2b03c55be94261d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d924da185d6d4bd7a2b03c55be94261d. Latest consensus state: current_term: 1 leader_uuid: "d924da185d6d4bd7a2b03c55be94261d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d924da185d6d4bd7a2b03c55be94261d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40917 } } }
I20260929 10:32:30.174834 8938 sys_catalog.cc:458] T 00000000000000000000000000000000 P d924da185d6d4bd7a2b03c55be94261d [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:30.175247 8945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:32:30.176496 8937 sys_catalog.cc:455] T 00000000000000000000000000000000 P d924da185d6d4bd7a2b03c55be94261d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d924da185d6d4bd7a2b03c55be94261d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d924da185d6d4bd7a2b03c55be94261d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40917 } } }
I20260929 10:32:30.176602 8937 sys_catalog.cc:458] T 00000000000000000000000000000000 P d924da185d6d4bd7a2b03c55be94261d [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:30.178468 8945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:32:30.184581 8945 catalog_manager.cc:1347] Generated new cluster ID: 1fef6b8c805b4a9a96e1b9d9420a3a63
I20260929 10:32:30.184671 8945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:32:30.196669 8945 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:32:30.197016 8945 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:32:30.208922 8945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d924da185d6d4bd7a2b03c55be94261d: Generated new TSK 0
I20260929 10:32:30.209736 8945 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:32:30.221069 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 8858
2026-09-29T10:32:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1431 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:32:30 dist-test-slave-9cj5 krb5kdc[8957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:32:30 dist-test-slave-9cj5 krb5kdc[8957](info): set up 2 sockets
Sep 29 10:32:30 dist-test-slave-9cj5 krb5kdc[8957](info): commencing operation
krb5kdc: starting...
W20260929 10:32:31.258589 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:32:31 dist-test-slave-9cj5 krb5kdc[8957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:32:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:32:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:32:31.385398 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46165
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:32:31.495301 8970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:31.495574 8970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:31.495616 8970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:31.499554 8970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:32:31.499616 8970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:31.499635 8970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:31.499652 8970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:32:31.499667 8970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:32:31.504909 8970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:31.505965 8970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:31.506943 8970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:31.515885 8978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:31.515931 8970 server_base.cc:1034] running on GCE node
W20260929 10:32:31.515851 8975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:31.515864 8976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:31.516666 8970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:31.517958 8970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:31.519189 8970 hybrid_clock.cc:648] HybridClock initialized: now 1790677951519154 us; error 46 us; skew 500 ppm
Sep 29 10:32:31 dist-test-slave-9cj5 krb5kdc[8957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:31.522399 8970 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:32:31.523507 8970 webserver.cc:460] Webserver started at http://127.0.0.1:36385/ using document root <none> and password file <none>
I20260929 10:32:31.524147 8970 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:31.524253 8970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:31.524523 8970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:31.526870 8970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "84c1909c7f0040e788ff73a59265b525"
format_stamp: "Formatted at 2026-09-29 10:32:31 on dist-test-slave-9cj5"
server_key: "505224f64be28be9f1ce6bfadc9d6c89"
server_key_iv: "77a95d91cda28e2e09578ac0577f719c"
server_key_version: "encryptionkey@0"
I20260929 10:32:31.527544 8970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "84c1909c7f0040e788ff73a59265b525"
format_stamp: "Formatted at 2026-09-29 10:32:31 on dist-test-slave-9cj5"
server_key: "505224f64be28be9f1ce6bfadc9d6c89"
server_key_iv: "77a95d91cda28e2e09578ac0577f719c"
server_key_version: "encryptionkey@0"
I20260929 10:32:31.532024 8970 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:32:31.534492 8985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:31.535732 8970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:32:31.535892 8970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "84c1909c7f0040e788ff73a59265b525"
format_stamp: "Formatted at 2026-09-29 10:32:31 on dist-test-slave-9cj5"
server_key: "505224f64be28be9f1ce6bfadc9d6c89"
server_key_iv: "77a95d91cda28e2e09578ac0577f719c"
server_key_version: "encryptionkey@0"
I20260929 10:32:31.536026 8970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:31.555228 8970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:31.558403 8970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:31.558627 8970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:31.566176 8970 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46165
I20260929 10:32:31.566203 9037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46165 every 8 connection(s)
I20260929 10:32:31.567137 8970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:32:31.569833 9038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:31.572014 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 8970
I20260929 10:32:31.572165 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:32:31.572458 20848 external_mini_cluster.cc:1428] Setting key 7a780edc61c8a1c3dbe441d0f6b746a3
I20260929 10:32:31.576085 9038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84c1909c7f0040e788ff73a59265b525: Bootstrap starting.
I20260929 10:32:31.578320 9038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84c1909c7f0040e788ff73a59265b525: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:31.579057 9038 log.cc:824] T 00000000000000000000000000000000 P 84c1909c7f0040e788ff73a59265b525: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:31.580996 9038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84c1909c7f0040e788ff73a59265b525: No bootstrap required, opened a new log
Sep 29 10:32:31 dist-test-slave-9cj5 krb5kdc[8957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 10:32:31.585834 9038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84c1909c7f0040e788ff73a59265b525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c1909c7f0040e788ff73a59265b525" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46165 } }
I20260929 10:32:31.586148 9038 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84c1909c7f0040e788ff73a59265b525 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:31.586238 9038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84c1909c7f0040e788ff73a59265b525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c1909c7f0040e788ff73a59265b525, State: Initialized, Role: FOLLOWER
I20260929 10:32:31.586828 9038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84c1909c7f0040e788ff73a59265b525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c1909c7f0040e788ff73a59265b525" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46165 } }
I20260929 10:32:31.586989 9038 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84c1909c7f0040e788ff73a59265b525 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:32:31.587071 9038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84c1909c7f0040e788ff73a59265b525 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:32:31.587203 9038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84c1909c7f0040e788ff73a59265b525 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:31.587641 9041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:31.573899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51639 (local address 127.0.0.1:46165)
0929 10:32:31.574405 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:31.574419 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:31.574453 (+ 34us) server_negotiation.cc:407] Connection header received
0929 10:32:31.575275 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:31.575293 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:31.575626 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:31.575960 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:31.576773 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:31.577731 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:31.578701 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:31.579198 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:31.579375 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:31.582060 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:31.582066 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:31.582083 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:31.582114 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:31.584196 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:31.584674 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:31.584680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:31.584687 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:31.584830 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:31.585271 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:31.585275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:31.585277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:31.585824 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:31.586049 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:31.586332 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 10:32:31.586673 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":129,"threads_started":1}
I20260929 10:32:31.588256 9038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84c1909c7f0040e788ff73a59265b525 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c1909c7f0040e788ff73a59265b525" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46165 } }
I20260929 10:32:31.588637 9038 leader_election.cc:302] T 00000000000000000000000000000000 P 84c1909c7f0040e788ff73a59265b525 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84c1909c7f0040e788ff73a59265b525; no voters:
I20260929 10:32:31.588990 9038 leader_election.cc:288] T 00000000000000000000000000000000 P 84c1909c7f0040e788ff73a59265b525 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:32:31.589167 9043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84c1909c7f0040e788ff73a59265b525 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:32:31.589449 9043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84c1909c7f0040e788ff73a59265b525 [term 1 LEADER]: Becoming Leader. State: Replica: 84c1909c7f0040e788ff73a59265b525, State: Running, Role: LEADER
I20260929 10:32:31.589866 9043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84c1909c7f0040e788ff73a59265b525 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c1909c7f0040e788ff73a59265b525" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46165 } }
I20260929 10:32:31.590297 9038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84c1909c7f0040e788ff73a59265b525 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:32:31.590751 20848 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:32:31.591495 9044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84c1909c7f0040e788ff73a59265b525 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84c1909c7f0040e788ff73a59265b525" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c1909c7f0040e788ff73a59265b525" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46165 } } }
I20260929 10:32:31.591634 9044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84c1909c7f0040e788ff73a59265b525 [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:31.592049 9045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84c1909c7f0040e788ff73a59265b525 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84c1909c7f0040e788ff73a59265b525. Latest consensus state: current_term: 1 leader_uuid: "84c1909c7f0040e788ff73a59265b525" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c1909c7f0040e788ff73a59265b525" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46165 } } }
I20260929 10:32:31.592216 9045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84c1909c7f0040e788ff73a59265b525 [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:31.593024 9052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:32:31.596048 9052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:32:31.602257 9052 catalog_manager.cc:1347] Generated new cluster ID: 1f0cd5daa9e148b6bfb763daee483f45
I20260929 10:32:31.602329 9052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:32:31.602959 9041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:31.593832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51705 (local address 127.0.0.1:46165)
0929 10:32:31.593983 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:31.593988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:31.594190 (+ 202us) server_negotiation.cc:407] Connection header received
0929 10:32:31.594314 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:31.594317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:31.594368 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:31.594470 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:31.595284 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:31.596130 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:31.597210 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:31.597609 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:31.597727 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:31.599192 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:31.599196 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:31.599199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:31.599224 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:31.601045 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:31.601554 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:31.601558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:31.601561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:31.601662 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:31.602002 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:31.602005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:31.602007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:31.602240 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:31.602340 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:31.602681 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 10:32:31.602803 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 10:32:31.646575 9052 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:32:31.647550 9052 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:32:31.656424 9052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84c1909c7f0040e788ff73a59265b525: Generated new TSK 0
I20260929 10:32:31.657157 9052 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:32:31.669333 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 8970
2026-09-29T10:32:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1447 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:32:31 dist-test-slave-9cj5 krb5kdc[9069](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:32:31 dist-test-slave-9cj5 krb5kdc[9069](info): set up 2 sockets
Sep 29 10:32:31 dist-test-slave-9cj5 krb5kdc[9069](info): commencing operation
krb5kdc: starting...
W20260929 10:32:32.727187 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:32:32 dist-test-slave-9cj5 krb5kdc[9069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:32:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:32:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:32:32.853792 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43901
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:32:32.964819 9082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:32.965092 9082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:32.965135 9082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:32.969197 9082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:32:32.969264 9082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:32.969283 9082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:32.969338 9082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:32:32.969376 9082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:32:32.975188 9082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:32.976197 9082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:32.977178 9082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:32.985539 9088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:32.985539 9087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:32.985625 9082 server_base.cc:1034] running on GCE node
W20260929 10:32:32.985555 9090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:32.986388 9082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:32.987468 9082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:32.988639 9082 hybrid_clock.cc:648] HybridClock initialized: now 1790677952988627 us; error 31 us; skew 500 ppm
Sep 29 10:32:32 dist-test-slave-9cj5 krb5kdc[9069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:32.991525 9082 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:32:32.992311 9082 webserver.cc:460] Webserver started at http://127.0.0.1:46331/ using document root <none> and password file <none>
I20260929 10:32:32.992820 9082 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:32.992878 9082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:32.993037 9082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:32.995075 9082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "da2259459bc944a2917821fb9648cfea"
format_stamp: "Formatted at 2026-09-29 10:32:32 on dist-test-slave-9cj5"
server_key: "435df1a9fe4656e4f3b3b0f4c8a33feb"
server_key_iv: "723f9aa2a5d2333e4b71956d00b58944"
server_key_version: "encryptionkey@0"
I20260929 10:32:32.995568 9082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "da2259459bc944a2917821fb9648cfea"
format_stamp: "Formatted at 2026-09-29 10:32:32 on dist-test-slave-9cj5"
server_key: "435df1a9fe4656e4f3b3b0f4c8a33feb"
server_key_iv: "723f9aa2a5d2333e4b71956d00b58944"
server_key_version: "encryptionkey@0"
I20260929 10:32:32.999212 9082 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:32:33.001555 9097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:33.002897 9082 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:32:33.003052 9082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "da2259459bc944a2917821fb9648cfea"
format_stamp: "Formatted at 2026-09-29 10:32:32 on dist-test-slave-9cj5"
server_key: "435df1a9fe4656e4f3b3b0f4c8a33feb"
server_key_iv: "723f9aa2a5d2333e4b71956d00b58944"
server_key_version: "encryptionkey@0"
I20260929 10:32:33.003163 9082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:33.015039 9082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:33.018271 9082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:33.018478 9082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:33.025312 9082 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43901
I20260929 10:32:33.025408 9149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43901 every 8 connection(s)
I20260929 10:32:33.026345 9082 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:32:33.029090 9150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:33.030575 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 9082
I20260929 10:32:33.030716 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:32:33.031010 20848 external_mini_cluster.cc:1428] Setting key 6977db83d46c7cced9999adee28915c1
I20260929 10:32:33.035939 9150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da2259459bc944a2917821fb9648cfea: Bootstrap starting.
I20260929 10:32:33.038525 9150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da2259459bc944a2917821fb9648cfea: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:33.039471 9150 log.cc:824] T 00000000000000000000000000000000 P da2259459bc944a2917821fb9648cfea: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:33.041157 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:33.032390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33677 (local address 127.0.0.1:43901)
0929 10:32:33.032962 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:33.032981 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:33.033023 (+ 42us) server_negotiation.cc:407] Connection header received
0929 10:32:33.033897 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:33.033920 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:33.034357 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:33.034725 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:33.035132 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:33.036547 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:33.037544 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:33.038230 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:33.038402 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:33.039282 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:33.039288 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:33.039303 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:33.039799 (+ 496us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:33.039869 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:33.039902 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:33.039999 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:33.040189 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 10:32:33.040426 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":170,"threads_started":1}
I20260929 10:32:33.041615 9150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da2259459bc944a2917821fb9648cfea: No bootstrap required, opened a new log
I20260929 10:32:33.043926 20848 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:32:33.045768 9150 raft_consensus.cc:348] T 00000000000000000000000000000000 P da2259459bc944a2917821fb9648cfea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2259459bc944a2917821fb9648cfea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43901 } }
I20260929 10:32:33.046048 9150 raft_consensus.cc:374] T 00000000000000000000000000000000 P da2259459bc944a2917821fb9648cfea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:33.046134 9150 raft_consensus.cc:729] T 00000000000000000000000000000000 P da2259459bc944a2917821fb9648cfea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da2259459bc944a2917821fb9648cfea, State: Initialized, Role: FOLLOWER
I20260929 10:32:33.046577 9150 consensus_queue.cc:260] T 00000000000000000000000000000000 P da2259459bc944a2917821fb9648cfea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2259459bc944a2917821fb9648cfea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43901 } }
I20260929 10:32:33.046728 9150 raft_consensus.cc:388] T 00000000000000000000000000000000 P da2259459bc944a2917821fb9648cfea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:32:33.046814 9150 raft_consensus.cc:482] T 00000000000000000000000000000000 P da2259459bc944a2917821fb9648cfea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:32:33.046937 9150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da2259459bc944a2917821fb9648cfea [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:33.047880 9150 raft_consensus.cc:504] T 00000000000000000000000000000000 P da2259459bc944a2917821fb9648cfea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2259459bc944a2917821fb9648cfea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43901 } }
I20260929 10:32:33.048226 9150 leader_election.cc:302] T 00000000000000000000000000000000 P da2259459bc944a2917821fb9648cfea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da2259459bc944a2917821fb9648cfea; no voters:
I20260929 10:32:33.048523 9150 leader_election.cc:288] T 00000000000000000000000000000000 P da2259459bc944a2917821fb9648cfea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:32:33.048676 9160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da2259459bc944a2917821fb9648cfea [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:32:33.048938 9160 raft_consensus.cc:686] T 00000000000000000000000000000000 P da2259459bc944a2917821fb9648cfea [term 1 LEADER]: Becoming Leader. State: Replica: da2259459bc944a2917821fb9648cfea, State: Running, Role: LEADER
I20260929 10:32:33.049276 9160 consensus_queue.cc:237] T 00000000000000000000000000000000 P da2259459bc944a2917821fb9648cfea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2259459bc944a2917821fb9648cfea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43901 } }
I20260929 10:32:33.049834 9150 sys_catalog.cc:564] T 00000000000000000000000000000000 P da2259459bc944a2917821fb9648cfea [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:32:33.050575 9162 sys_catalog.cc:455] T 00000000000000000000000000000000 P da2259459bc944a2917821fb9648cfea [sys.catalog]: SysCatalogTable state changed. Reason: New leader da2259459bc944a2917821fb9648cfea. Latest consensus state: current_term: 1 leader_uuid: "da2259459bc944a2917821fb9648cfea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2259459bc944a2917821fb9648cfea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43901 } } }
I20260929 10:32:33.050693 9162 sys_catalog.cc:458] T 00000000000000000000000000000000 P da2259459bc944a2917821fb9648cfea [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:33.051052 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:33.046239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33329 (local address 127.0.0.1:43901)
0929 10:32:33.046350 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:33.046355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:33.046750 (+ 395us) server_negotiation.cc:407] Connection header received
0929 10:32:33.046887 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:33.046890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:33.046938 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:33.047028 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:33.047311 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:33.047829 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:33.048842 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:33.049207 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:33.049284 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:33.050292 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:33.050296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:33.050299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:33.050312 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:33.050342 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:33.050355 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:33.050433 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:33.050594 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 10:32:33.050661 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 10:32:33.051122 9168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:32:33.052747 9161 sys_catalog.cc:455] T 00000000000000000000000000000000 P da2259459bc944a2917821fb9648cfea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da2259459bc944a2917821fb9648cfea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2259459bc944a2917821fb9648cfea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43901 } } }
I20260929 10:32:33.052846 9161 sys_catalog.cc:458] T 00000000000000000000000000000000 P da2259459bc944a2917821fb9648cfea [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:33.054180 9168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:32:33.059513 9168 catalog_manager.cc:1347] Generated new cluster ID: e3eee4a8beca464db48cb82520a8161b
I20260929 10:32:33.059568 9168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:32:33.070497 9168 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:32:33.071420 9168 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:32:33.075814 9168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da2259459bc944a2917821fb9648cfea: Generated new TSK 0
I20260929 10:32:33.076740 9168 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:32:33.082460 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 9082
2026-09-29T10:32:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1413 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:32:33 dist-test-slave-9cj5 krb5kdc[9181](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:32:33 dist-test-slave-9cj5 krb5kdc[9181](info): set up 2 sockets
Sep 29 10:32:33 dist-test-slave-9cj5 krb5kdc[9181](info): commencing operation
krb5kdc: starting...
W20260929 10:32:34.139662 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:32:34 dist-test-slave-9cj5 krb5kdc[9181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:32:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:32:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:32:34.266016 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38335
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:32:34.373980 9194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:34.374331 9194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:34.374415 9194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:34.378558 9194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:32:34.378650 9194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:34.378688 9194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:34.378746 9194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:32:34.378784 9194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:32:34.384691 9194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:34.385995 9194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:34.387094 9194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:34.395925 9199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:34.395919 9200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:34.395946 9202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:34.396317 9194 server_base.cc:1034] running on GCE node
I20260929 10:32:34.396857 9194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:34.398069 9194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:34.399240 9194 hybrid_clock.cc:648] HybridClock initialized: now 1790677954399209 us; error 40 us; skew 500 ppm
Sep 29 10:32:34 dist-test-slave-9cj5 krb5kdc[9181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:34.402343 9194 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:32:34.403261 9194 webserver.cc:460] Webserver started at http://127.0.0.1:34905/ using document root <none> and password file <none>
I20260929 10:32:34.403824 9194 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:34.403920 9194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:34.404131 9194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:34.406203 9194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "12cee01d90e54fa1ae3c7a504372aa5f"
format_stamp: "Formatted at 2026-09-29 10:32:34 on dist-test-slave-9cj5"
server_key: "fd8290c601bbf93708d10ae15c5a40d5"
server_key_iv: "32cefd45f9cca988c5b4d2f7443ce937"
server_key_version: "encryptionkey@0"
I20260929 10:32:34.406759 9194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "12cee01d90e54fa1ae3c7a504372aa5f"
format_stamp: "Formatted at 2026-09-29 10:32:34 on dist-test-slave-9cj5"
server_key: "fd8290c601bbf93708d10ae15c5a40d5"
server_key_iv: "32cefd45f9cca988c5b4d2f7443ce937"
server_key_version: "encryptionkey@0"
I20260929 10:32:34.410420 9194 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 10:32:34.412678 9209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:34.413873 9194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260929 10:32:34.414005 9194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "12cee01d90e54fa1ae3c7a504372aa5f"
format_stamp: "Formatted at 2026-09-29 10:32:34 on dist-test-slave-9cj5"
server_key: "fd8290c601bbf93708d10ae15c5a40d5"
server_key_iv: "32cefd45f9cca988c5b4d2f7443ce937"
server_key_version: "encryptionkey@0"
I20260929 10:32:34.414124 9194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:34.418646 9194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:34.421365 9194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:34.421564 9194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:34.428725 9194 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38335
I20260929 10:32:34.428743 9261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38335 every 8 connection(s)
I20260929 10:32:34.429651 9194 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:32:34.431906 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 9194
I20260929 10:32:34.432062 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:32:34.432356 20848 external_mini_cluster.cc:1428] Setting key d7a8baec2b91d31d22fb20cb76706aff
I20260929 10:32:34.432452 9262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:34.437661 9265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:34.433706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56079 (local address 127.0.0.1:38335)
0929 10:32:34.434181 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:34.434196 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:34.434230 (+ 34us) server_negotiation.cc:407] Connection header received
0929 10:32:34.434957 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:34.434972 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:34.435242 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:34.435509 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:34.436340 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:34.436347 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:34.436361 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:34.436374 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:34.436438 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:34.436463 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:34.436602 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:34.436667 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 10:32:34.436805 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":137,"threads_started":1}
I20260929 10:32:34.439254 9262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12cee01d90e54fa1ae3c7a504372aa5f: Bootstrap starting.
I20260929 10:32:34.440397 20848 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:32:34.441514 9262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12cee01d90e54fa1ae3c7a504372aa5f: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:34.442363 9262 log.cc:824] T 00000000000000000000000000000000 P 12cee01d90e54fa1ae3c7a504372aa5f: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:34.444182 9265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:34.442549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43293 (local address 127.0.0.1:38335)
0929 10:32:34.442655 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:34.442659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:34.442824 (+ 165us) server_negotiation.cc:407] Connection header received
0929 10:32:34.442968 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:34.442971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:34.443010 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:34.443095 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:34.443791 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:34.443795 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:34.443798 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:34.443809 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:34.443835 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:34.443848 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:34.443943 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:34.444016 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 10:32:34.444062 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 10:32:34.444415 9262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12cee01d90e54fa1ae3c7a504372aa5f: No bootstrap required, opened a new log
I20260929 10:32:34.448417 9262 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12cee01d90e54fa1ae3c7a504372aa5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12cee01d90e54fa1ae3c7a504372aa5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38335 } }
I20260929 10:32:34.448694 9262 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12cee01d90e54fa1ae3c7a504372aa5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:34.448761 9262 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12cee01d90e54fa1ae3c7a504372aa5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12cee01d90e54fa1ae3c7a504372aa5f, State: Initialized, Role: FOLLOWER
I20260929 10:32:34.449200 9262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12cee01d90e54fa1ae3c7a504372aa5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12cee01d90e54fa1ae3c7a504372aa5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38335 } }
I20260929 10:32:34.449324 9262 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12cee01d90e54fa1ae3c7a504372aa5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:32:34.449365 9262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12cee01d90e54fa1ae3c7a504372aa5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:32:34.449429 9262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12cee01d90e54fa1ae3c7a504372aa5f [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:34.450379 9262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12cee01d90e54fa1ae3c7a504372aa5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12cee01d90e54fa1ae3c7a504372aa5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38335 } }
I20260929 10:32:34.450677 9262 leader_election.cc:302] T 00000000000000000000000000000000 P 12cee01d90e54fa1ae3c7a504372aa5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12cee01d90e54fa1ae3c7a504372aa5f; no voters:
I20260929 10:32:34.450965 9262 leader_election.cc:288] T 00000000000000000000000000000000 P 12cee01d90e54fa1ae3c7a504372aa5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:32:34.451097 9272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12cee01d90e54fa1ae3c7a504372aa5f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:32:34.451321 9272 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12cee01d90e54fa1ae3c7a504372aa5f [term 1 LEADER]: Becoming Leader. State: Replica: 12cee01d90e54fa1ae3c7a504372aa5f, State: Running, Role: LEADER
I20260929 10:32:34.451654 9272 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12cee01d90e54fa1ae3c7a504372aa5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12cee01d90e54fa1ae3c7a504372aa5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38335 } }
I20260929 10:32:34.452162 9262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12cee01d90e54fa1ae3c7a504372aa5f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:32:34.453293 9274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12cee01d90e54fa1ae3c7a504372aa5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12cee01d90e54fa1ae3c7a504372aa5f. Latest consensus state: current_term: 1 leader_uuid: "12cee01d90e54fa1ae3c7a504372aa5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12cee01d90e54fa1ae3c7a504372aa5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38335 } } }
I20260929 10:32:34.453428 9274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12cee01d90e54fa1ae3c7a504372aa5f [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:34.453732 9273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12cee01d90e54fa1ae3c7a504372aa5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12cee01d90e54fa1ae3c7a504372aa5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12cee01d90e54fa1ae3c7a504372aa5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38335 } } }
I20260929 10:32:34.453842 9273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12cee01d90e54fa1ae3c7a504372aa5f [sys.catalog]: This master's current role is: LEADER
W20260929 10:32:34.456924 9287 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 12cee01d90e54fa1ae3c7a504372aa5f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:32:34.456990 9287 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:32:34.457072 9281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:32:34.457815 9281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:32:34.463092 9281 catalog_manager.cc:1347] Generated new cluster ID: 2c2a3f8a4fc14039bd74197086997ba7
I20260929 10:32:34.463183 9281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:32:34.473526 9281 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:32:34.473905 9281 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:32:34.485221 9281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12cee01d90e54fa1ae3c7a504372aa5f: Generated new TSK 0
I20260929 10:32:34.486021 9281 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:32:34.498836 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 9194
2026-09-29T10:32:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:32:34 dist-test-slave-9cj5 krb5kdc[9293](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:32:34 dist-test-slave-9cj5 krb5kdc[9293](info): set up 2 sockets
Sep 29 10:32:34 dist-test-slave-9cj5 krb5kdc[9293](info): commencing operation
krb5kdc: starting...
W20260929 10:32:35.538740 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:32:35 dist-test-slave-9cj5 krb5kdc[9293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:32:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:32:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:32:35.665519 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39935
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:32:35.777565 9306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:35.777937 9306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:35.777985 9306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:35.782214 9306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:32:35.782284 9306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:35.782303 9306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:35.782320 9306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:32:35.782336 9306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:32:35.787921 9306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:35.789047 9306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:35.790117 9306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:32:35.798633 9306 server_base.cc:1034] running on GCE node
W20260929 10:32:35.798616 9311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:35.798616 9312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:35.798902 9314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:35.799507 9306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:35.800822 9306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:35.802026 9306 hybrid_clock.cc:648] HybridClock initialized: now 1790677955801993 us; error 57 us; skew 500 ppm
Sep 29 10:32:35 dist-test-slave-9cj5 krb5kdc[9293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:35.809309 9306 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:32:35.810397 9306 webserver.cc:460] Webserver started at http://127.0.0.1:34431/ using document root <none> and password file <none>
I20260929 10:32:35.811007 9306 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:35.811105 9306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:35.811353 9306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:35.813442 9306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "88465b2ce7e84578b9483e66cda3ce80"
format_stamp: "Formatted at 2026-09-29 10:32:35 on dist-test-slave-9cj5"
server_key: "46fca6c326f050d8d6c94bdcc820ff35"
server_key_iv: "549e8bd43ded0e3e85c02a50515106fd"
server_key_version: "encryptionkey@0"
I20260929 10:32:35.814042 9306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "88465b2ce7e84578b9483e66cda3ce80"
format_stamp: "Formatted at 2026-09-29 10:32:35 on dist-test-slave-9cj5"
server_key: "46fca6c326f050d8d6c94bdcc820ff35"
server_key_iv: "549e8bd43ded0e3e85c02a50515106fd"
server_key_version: "encryptionkey@0"
I20260929 10:32:35.817961 9306 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 10:32:35.820391 9321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:35.821583 9306 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:32:35.821806 9306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "88465b2ce7e84578b9483e66cda3ce80"
format_stamp: "Formatted at 2026-09-29 10:32:35 on dist-test-slave-9cj5"
server_key: "46fca6c326f050d8d6c94bdcc820ff35"
server_key_iv: "549e8bd43ded0e3e85c02a50515106fd"
server_key_version: "encryptionkey@0"
I20260929 10:32:35.821931 9306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:35.839761 9306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:35.843189 9306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:35.843407 9306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:35.850775 9306 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39935
I20260929 10:32:35.850802 9373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39935 every 8 connection(s)
I20260929 10:32:35.851750 9306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:32:35.852559 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 9306
I20260929 10:32:35.852694 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:32:35.853003 20848 external_mini_cluster.cc:1428] Setting key 6cd68ce90cda7af2fce361f6e20ad51f
I20260929 10:32:35.855085 9374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:35.860549 9374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88465b2ce7e84578b9483e66cda3ce80: Bootstrap starting.
Sep 29 10:32:35 dist-test-slave-9cj5 krb5kdc[9293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 10:32:35.863032 9374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88465b2ce7e84578b9483e66cda3ce80: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:35.863917 9374 log.cc:824] T 00000000000000000000000000000000 P 88465b2ce7e84578b9483e66cda3ce80: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:35.865856 9374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88465b2ce7e84578b9483e66cda3ce80: No bootstrap required, opened a new log
I20260929 10:32:35.867678 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:35.854455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:57257 (local address 127.0.0.1:39935)
0929 10:32:35.854886 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:35.854900 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:35.854935 (+ 35us) server_negotiation.cc:407] Connection header received
0929 10:32:35.855626 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:35.855643 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:35.855979 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:35.856323 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:35.857136 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:35.858355 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:35.859248 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:35.859726 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:35.859901 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:35.862813 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:35.862819 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:35.862832 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:35.862862 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:35.864671 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:35.865206 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:35.865211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:35.865215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:35.865325 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:35.865676 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:35.865682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:35.865688 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:35.866184 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:35.866416 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:35.866734 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 10:32:35.866960 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":148,"threads_started":1}
I20260929 10:32:35.870146 9374 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88465b2ce7e84578b9483e66cda3ce80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88465b2ce7e84578b9483e66cda3ce80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39935 } }
I20260929 10:32:35.870451 9374 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88465b2ce7e84578b9483e66cda3ce80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:35.870527 9374 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88465b2ce7e84578b9483e66cda3ce80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88465b2ce7e84578b9483e66cda3ce80, State: Initialized, Role: FOLLOWER
I20260929 10:32:35.871037 9374 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88465b2ce7e84578b9483e66cda3ce80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88465b2ce7e84578b9483e66cda3ce80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39935 } }
I20260929 10:32:35.871206 9374 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88465b2ce7e84578b9483e66cda3ce80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:32:35.871230 20848 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:32:35.871290 9374 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88465b2ce7e84578b9483e66cda3ce80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:32:35.871390 9374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88465b2ce7e84578b9483e66cda3ce80 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:35.872376 9374 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88465b2ce7e84578b9483e66cda3ce80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88465b2ce7e84578b9483e66cda3ce80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39935 } }
I20260929 10:32:35.872735 9374 leader_election.cc:302] T 00000000000000000000000000000000 P 88465b2ce7e84578b9483e66cda3ce80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88465b2ce7e84578b9483e66cda3ce80; no voters:
I20260929 10:32:35.873099 9374 leader_election.cc:288] T 00000000000000000000000000000000 P 88465b2ce7e84578b9483e66cda3ce80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:32:35.873181 9383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88465b2ce7e84578b9483e66cda3ce80 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:32:35.873395 9383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88465b2ce7e84578b9483e66cda3ce80 [term 1 LEADER]: Becoming Leader. State: Replica: 88465b2ce7e84578b9483e66cda3ce80, State: Running, Role: LEADER
I20260929 10:32:35.873697 9383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88465b2ce7e84578b9483e66cda3ce80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88465b2ce7e84578b9483e66cda3ce80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39935 } }
I20260929 10:32:35.874480 9374 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88465b2ce7e84578b9483e66cda3ce80 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:32:35.875317 9384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88465b2ce7e84578b9483e66cda3ce80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88465b2ce7e84578b9483e66cda3ce80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88465b2ce7e84578b9483e66cda3ce80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39935 } } }
I20260929 10:32:35.875455 9384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88465b2ce7e84578b9483e66cda3ce80 [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:35.875689 9384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88465b2ce7e84578b9483e66cda3ce80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88465b2ce7e84578b9483e66cda3ce80. Latest consensus state: current_term: 1 leader_uuid: "88465b2ce7e84578b9483e66cda3ce80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88465b2ce7e84578b9483e66cda3ce80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39935 } } }
I20260929 10:32:35.875782 9384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88465b2ce7e84578b9483e66cda3ce80 [sys.catalog]: This master's current role is: LEADER
W20260929 10:32:35.879613 9399 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88465b2ce7e84578b9483e66cda3ce80: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:32:35.879717 9399 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:32:35.879803 9393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:32:35.880713 9393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:32:35.883766 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:35.873355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:39043 (local address 127.0.0.1:39935)
0929 10:32:35.873502 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:35.873507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:35.874059 (+ 552us) server_negotiation.cc:407] Connection header received
0929 10:32:35.874212 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:35.874215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:35.874263 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:35.874360 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:35.875335 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:35.876215 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:35.877359 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:35.877668 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:35.877804 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:35.879537 (+ 1733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:35.879541 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:35.879544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:35.879571 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:35.881406 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:35.882085 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:35.882089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:35.882091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:35.882193 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:35.882568 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:35.882572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:35.882574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:35.882829 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:35.883015 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:35.883326 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 10:32:35.883601 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":47}
I20260929 10:32:35.886627 9393 catalog_manager.cc:1347] Generated new cluster ID: 529629987863491da9d19bde01dda2e0
I20260929 10:32:35.886690 9393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:32:35.897907 9393 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:32:35.899195 9393 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:32:35.911653 9393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88465b2ce7e84578b9483e66cda3ce80: Generated new TSK 0
I20260929 10:32:35.912523 9393 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:32:35.924615 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 9306
W20260929 10:32:35.930076 9291 connection.cc:535] client connection to 127.0.0.1:39935 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39935): Connection reset by peer (error 104)
2026-09-29T10:32:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1425 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:32:35 dist-test-slave-9cj5 krb5kdc[9405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:32:35 dist-test-slave-9cj5 krb5kdc[9405](info): set up 2 sockets
Sep 29 10:32:35 dist-test-slave-9cj5 krb5kdc[9405](info): commencing operation
krb5kdc: starting...
W20260929 10:32:36.970760 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:32:37 dist-test-slave-9cj5 krb5kdc[9405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:32:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:32:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:32:37.098274 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41193
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:32:37.213315 9418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:37.213609 9418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:37.213654 9418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:37.217885 9418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:32:37.217959 9418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:37.217980 9418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:37.217998 9418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:32:37.218015 9418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:32:37.223464 9418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:37.224696 9418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:37.225734 9418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:37.234458 9424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:37.234549 9426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:37.234452 9423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:37.234710 9418 server_base.cc:1034] running on GCE node
I20260929 10:32:37.235561 9418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:37.236761 9418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:37.237987 9418 hybrid_clock.cc:648] HybridClock initialized: now 1790677957237974 us; error 42 us; skew 500 ppm
Sep 29 10:32:37 dist-test-slave-9cj5 krb5kdc[9405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:37.241039 9418 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:32:37.242122 9418 webserver.cc:460] Webserver started at http://127.0.0.1:40475/ using document root <none> and password file <none>
I20260929 10:32:37.242718 9418 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:37.242868 9418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:37.243153 9418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:37.245396 9418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "db516dae954648a4b5902d9e7230b206"
format_stamp: "Formatted at 2026-09-29 10:32:37 on dist-test-slave-9cj5"
server_key: "b841a3d14c1afd641ebcf3f74341112f"
server_key_iv: "c548b97c95dadb646420f64152e8a062"
server_key_version: "encryptionkey@0"
I20260929 10:32:37.246052 9418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "db516dae954648a4b5902d9e7230b206"
format_stamp: "Formatted at 2026-09-29 10:32:37 on dist-test-slave-9cj5"
server_key: "b841a3d14c1afd641ebcf3f74341112f"
server_key_iv: "c548b97c95dadb646420f64152e8a062"
server_key_version: "encryptionkey@0"
I20260929 10:32:37.250173 9418 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 10:32:37.252785 9433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:37.254244 9418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:32:37.254401 9418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "db516dae954648a4b5902d9e7230b206"
format_stamp: "Formatted at 2026-09-29 10:32:37 on dist-test-slave-9cj5"
server_key: "b841a3d14c1afd641ebcf3f74341112f"
server_key_iv: "c548b97c95dadb646420f64152e8a062"
server_key_version: "encryptionkey@0"
I20260929 10:32:37.254524 9418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:37.274537 9418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:37.277715 9418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:37.277944 9418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:37.284698 9418 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41193
I20260929 10:32:37.284727 9485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41193 every 8 connection(s)
I20260929 10:32:37.285627 9418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:32:37.288395 9486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:37.294145 9486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db516dae954648a4b5902d9e7230b206: Bootstrap starting.
I20260929 10:32:37.295348 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 9418
I20260929 10:32:37.295517 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:32:37.295840 20848 external_mini_cluster.cc:1428] Setting key 926b89fb6630d74e3496d9dd696b3b05
I20260929 10:32:37.297243 9486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db516dae954648a4b5902d9e7230b206: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:37.298063 9486 log.cc:824] T 00000000000000000000000000000000 P db516dae954648a4b5902d9e7230b206: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:37.300011 9486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db516dae954648a4b5902d9e7230b206: No bootstrap required, opened a new log
I20260929 10:32:37.303681 9486 raft_consensus.cc:348] T 00000000000000000000000000000000 P db516dae954648a4b5902d9e7230b206 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db516dae954648a4b5902d9e7230b206" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41193 } }
I20260929 10:32:37.303931 9486 raft_consensus.cc:374] T 00000000000000000000000000000000 P db516dae954648a4b5902d9e7230b206 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:37.303990 9486 raft_consensus.cc:729] T 00000000000000000000000000000000 P db516dae954648a4b5902d9e7230b206 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db516dae954648a4b5902d9e7230b206, State: Initialized, Role: FOLLOWER
I20260929 10:32:37.304459 9486 consensus_queue.cc:260] T 00000000000000000000000000000000 P db516dae954648a4b5902d9e7230b206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db516dae954648a4b5902d9e7230b206" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41193 } }
I20260929 10:32:37.304632 9486 raft_consensus.cc:388] T 00000000000000000000000000000000 P db516dae954648a4b5902d9e7230b206 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:32:37.304714 9486 raft_consensus.cc:482] T 00000000000000000000000000000000 P db516dae954648a4b5902d9e7230b206 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:32:37.304838 9486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db516dae954648a4b5902d9e7230b206 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:37.305037 9489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:37.297033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:37213 (local address 127.0.0.1:41193)
0929 10:32:37.297524 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:37.297538 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:37.297572 (+ 34us) server_negotiation.cc:407] Connection header received
0929 10:32:37.298350 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:37.298368 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:37.298813 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:37.299182 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:37.299586 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:37.300600 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:37.301482 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:37.302047 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:37.302226 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:37.303176 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:37.303180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:37.303193 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:37.303576 (+ 383us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:37.303630 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:37.303652 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:37.303794 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:37.303973 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 10:32:37.304118 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":133,"threads_started":1}
I20260929 10:32:37.306015 9486 raft_consensus.cc:504] T 00000000000000000000000000000000 P db516dae954648a4b5902d9e7230b206 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db516dae954648a4b5902d9e7230b206" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41193 } }
I20260929 10:32:37.306342 9486 leader_election.cc:302] T 00000000000000000000000000000000 P db516dae954648a4b5902d9e7230b206 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db516dae954648a4b5902d9e7230b206; no voters:
I20260929 10:32:37.306687 9486 leader_election.cc:288] T 00000000000000000000000000000000 P db516dae954648a4b5902d9e7230b206 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:32:37.306754 9491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db516dae954648a4b5902d9e7230b206 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:32:37.307303 9491 raft_consensus.cc:686] T 00000000000000000000000000000000 P db516dae954648a4b5902d9e7230b206 [term 1 LEADER]: Becoming Leader. State: Replica: db516dae954648a4b5902d9e7230b206, State: Running, Role: LEADER
I20260929 10:32:37.307611 9491 consensus_queue.cc:237] T 00000000000000000000000000000000 P db516dae954648a4b5902d9e7230b206 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db516dae954648a4b5902d9e7230b206" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41193 } }
I20260929 10:32:37.307989 9486 sys_catalog.cc:564] T 00000000000000000000000000000000 P db516dae954648a4b5902d9e7230b206 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:32:37.308274 20848 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:32:37.309288 9492 sys_catalog.cc:455] T 00000000000000000000000000000000 P db516dae954648a4b5902d9e7230b206 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db516dae954648a4b5902d9e7230b206" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db516dae954648a4b5902d9e7230b206" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41193 } } }
I20260929 10:32:37.309412 9492 sys_catalog.cc:458] T 00000000000000000000000000000000 P db516dae954648a4b5902d9e7230b206 [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:37.309716 9493 sys_catalog.cc:455] T 00000000000000000000000000000000 P db516dae954648a4b5902d9e7230b206 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db516dae954648a4b5902d9e7230b206. Latest consensus state: current_term: 1 leader_uuid: "db516dae954648a4b5902d9e7230b206" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db516dae954648a4b5902d9e7230b206" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41193 } } }
I20260929 10:32:37.309865 9493 sys_catalog.cc:458] T 00000000000000000000000000000000 P db516dae954648a4b5902d9e7230b206 [sys.catalog]: This master's current role is: LEADER
W20260929 10:32:37.313678 9510 catalog_manager.cc:1558] T 00000000000000000000000000000000 P db516dae954648a4b5902d9e7230b206: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:32:37.313794 9510 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:32:37.313910 9503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:32:37.314735 9503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:32:37.314896 9489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:37.310691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:50195 (local address 127.0.0.1:41193)
0929 10:32:37.310788 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:37.310793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:37.311229 (+ 436us) server_negotiation.cc:407] Connection header received
0929 10:32:37.311335 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:37.311337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:37.311377 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:37.311441 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:37.311734 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:37.312244 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:37.313101 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:37.313346 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:37.313456 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:37.314341 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:37.314345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:37.314347 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:37.314361 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:37.314393 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:37.314404 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:37.314499 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:37.314677 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 10:32:37.314710 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 10:32:37.320602 9503 catalog_manager.cc:1347] Generated new cluster ID: 5b6a62ee715a42a2a6571a392db8aebd
I20260929 10:32:37.320708 9503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:32:37.334537 9503 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:32:37.335466 9503 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:32:37.343079 9503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db516dae954648a4b5902d9e7230b206: Generated new TSK 0
I20260929 10:32:37.343843 9503 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:32:37.352527 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 9418
W20260929 10:32:37.357767 9403 connection.cc:535] client connection to 127.0.0.1:41193 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41193): Connection reset by peer (error 104)
2026-09-29T10:32:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1427 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:32:37 dist-test-slave-9cj5 krb5kdc[9517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:32:37 dist-test-slave-9cj5 krb5kdc[9517](info): set up 2 sockets
Sep 29 10:32:37 dist-test-slave-9cj5 krb5kdc[9517](info): commencing operation
krb5kdc: starting...
W20260929 10:32:38.386858 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:32:38 dist-test-slave-9cj5 krb5kdc[9517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:32:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:32:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:32:38.512413 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37875
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:32:38.621517 9530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:38.621896 9530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:38.621946 9530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:38.625968 9530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:32:38.626024 9530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:38.626044 9530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:38.626060 9530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:32:38.626076 9530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:32:38.631445 9530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:38.632534 9530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:38.633507 9530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:38.642186 9535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:38.642238 9536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:38.642318 9530 server_base.cc:1034] running on GCE node
W20260929 10:32:38.642244 9538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:38.643026 9530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:38.644085 9530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:38.645242 9530 hybrid_clock.cc:648] HybridClock initialized: now 1790677958645215 us; error 41 us; skew 500 ppm
Sep 29 10:32:38 dist-test-slave-9cj5 krb5kdc[9517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:38.648010 9530 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:32:38.648890 9530 webserver.cc:460] Webserver started at http://127.0.0.1:37379/ using document root <none> and password file <none>
I20260929 10:32:38.649384 9530 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:38.649438 9530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:38.649593 9530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:38.651611 9530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "5a5c44bca24248c7971f49f378f4148e"
format_stamp: "Formatted at 2026-09-29 10:32:38 on dist-test-slave-9cj5"
server_key: "fd207ac1db2ef738ff6f30533093f7ef"
server_key_iv: "f68dd0089f75e9005695780d1c30bbe1"
server_key_version: "encryptionkey@0"
I20260929 10:32:38.652120 9530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "5a5c44bca24248c7971f49f378f4148e"
format_stamp: "Formatted at 2026-09-29 10:32:38 on dist-test-slave-9cj5"
server_key: "fd207ac1db2ef738ff6f30533093f7ef"
server_key_iv: "f68dd0089f75e9005695780d1c30bbe1"
server_key_version: "encryptionkey@0"
I20260929 10:32:38.655855 9530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:32:38.658219 9545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:38.659428 9530 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:32:38.659525 9530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "5a5c44bca24248c7971f49f378f4148e"
format_stamp: "Formatted at 2026-09-29 10:32:38 on dist-test-slave-9cj5"
server_key: "fd207ac1db2ef738ff6f30533093f7ef"
server_key_iv: "f68dd0089f75e9005695780d1c30bbe1"
server_key_version: "encryptionkey@0"
I20260929 10:32:38.659610 9530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:38.664077 9530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:38.667001 9530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:38.667160 9530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:38.674521 9530 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37875
I20260929 10:32:38.674628 9597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37875 every 8 connection(s)
I20260929 10:32:38.675485 9530 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:32:38.678305 9598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:38.678573 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 9530
I20260929 10:32:38.678707 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:32:38.678985 20848 external_mini_cluster.cc:1428] Setting key d70a50ebf104dd12d5451a791ab9ddc5
I20260929 10:32:38.684188 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:38.680357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:47961 (local address 127.0.0.1:37875)
0929 10:32:38.680804 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:38.680819 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:38.680852 (+ 33us) server_negotiation.cc:407] Connection header received
0929 10:32:38.681529 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:38.681544 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:38.681892 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:38.682231 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:38.683046 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:38.683051 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:38.683062 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:38.683074 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:38.683126 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:38.683145 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:38.683298 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:38.683366 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 10:32:38.683522 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":164,"threads_started":1}
I20260929 10:32:38.685245 9598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a5c44bca24248c7971f49f378f4148e: Bootstrap starting.
I20260929 10:32:38.687037 20848 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:32:38.687785 9598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a5c44bca24248c7971f49f378f4148e: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:38.688573 9598 log.cc:824] T 00000000000000000000000000000000 P 5a5c44bca24248c7971f49f378f4148e: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:38.691008 9598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a5c44bca24248c7971f49f378f4148e: No bootstrap required, opened a new log
I20260929 10:32:38.691228 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:38.689172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:35409 (local address 127.0.0.1:37875)
0929 10:32:38.689286 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:38.689290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:38.689576 (+ 286us) server_negotiation.cc:407] Connection header received
0929 10:32:38.689698 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:38.689700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:38.689730 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:38.689855 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:38.690744 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:38.690747 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:38.690749 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:38.690760 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:38.690783 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:38.690812 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:38.690919 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:38.691031 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 10:32:38.691098 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 10:32:38.694730 9598 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a5c44bca24248c7971f49f378f4148e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5c44bca24248c7971f49f378f4148e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37875 } }
I20260929 10:32:38.694950 9598 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a5c44bca24248c7971f49f378f4148e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:38.695024 9598 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a5c44bca24248c7971f49f378f4148e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a5c44bca24248c7971f49f378f4148e, State: Initialized, Role: FOLLOWER
I20260929 10:32:38.695497 9598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a5c44bca24248c7971f49f378f4148e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5c44bca24248c7971f49f378f4148e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37875 } }
I20260929 10:32:38.695631 9598 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a5c44bca24248c7971f49f378f4148e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:32:38.695711 9598 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a5c44bca24248c7971f49f378f4148e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:32:38.695828 9598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a5c44bca24248c7971f49f378f4148e [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:38.696923 9598 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a5c44bca24248c7971f49f378f4148e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5c44bca24248c7971f49f378f4148e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37875 } }
I20260929 10:32:38.697279 9598 leader_election.cc:302] T 00000000000000000000000000000000 P 5a5c44bca24248c7971f49f378f4148e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a5c44bca24248c7971f49f378f4148e; no voters:
I20260929 10:32:38.697628 9598 leader_election.cc:288] T 00000000000000000000000000000000 P 5a5c44bca24248c7971f49f378f4148e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:32:38.697738 9608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a5c44bca24248c7971f49f378f4148e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:32:38.698021 9608 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a5c44bca24248c7971f49f378f4148e [term 1 LEADER]: Becoming Leader. State: Replica: 5a5c44bca24248c7971f49f378f4148e, State: Running, Role: LEADER
I20260929 10:32:38.698395 9608 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a5c44bca24248c7971f49f378f4148e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5c44bca24248c7971f49f378f4148e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37875 } }
I20260929 10:32:38.698951 9598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a5c44bca24248c7971f49f378f4148e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:32:38.700120 9610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a5c44bca24248c7971f49f378f4148e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a5c44bca24248c7971f49f378f4148e. Latest consensus state: current_term: 1 leader_uuid: "5a5c44bca24248c7971f49f378f4148e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5c44bca24248c7971f49f378f4148e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37875 } } }
I20260929 10:32:38.700167 9609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a5c44bca24248c7971f49f378f4148e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a5c44bca24248c7971f49f378f4148e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5c44bca24248c7971f49f378f4148e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37875 } } }
I20260929 10:32:38.700243 9610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a5c44bca24248c7971f49f378f4148e [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:38.700261 9609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a5c44bca24248c7971f49f378f4148e [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:38.700608 9617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:32:38.703761 9617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:32:38.709287 9617 catalog_manager.cc:1347] Generated new cluster ID: cd44be74452740d0b0c12d5500f7f1a1
I20260929 10:32:38.709439 9617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:32:38.732824 9617 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:32:38.733199 9617 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:32:38.749128 9617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a5c44bca24248c7971f49f378f4148e: Generated new TSK 0
I20260929 10:32:38.750058 9617 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:32:38.758689 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 9530
2026-09-29T10:32:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:32:38 dist-test-slave-9cj5 krb5kdc[9629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:32:38 dist-test-slave-9cj5 krb5kdc[9629](info): set up 2 sockets
Sep 29 10:32:38 dist-test-slave-9cj5 krb5kdc[9629](info): commencing operation
krb5kdc: starting...
W20260929 10:32:39.822745 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:32:39 dist-test-slave-9cj5 krb5kdc[9629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:32:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:32:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:32:39.948566 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36685
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:32:40.058504 9642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:40.058784 9642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:40.058826 9642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:40.062858 9642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:32:40.062920 9642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:40.062940 9642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:40.062958 9642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:32:40.062974 9642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:32:40.068353 9642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36685
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:40.069365 9642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:40.070343 9642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:40.080021 9647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:40.079926 9650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:40.079926 9648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:40.080147 9642 server_base.cc:1034] running on GCE node
I20260929 10:32:40.080927 9642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:40.082113 9642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:40.083285 9642 hybrid_clock.cc:648] HybridClock initialized: now 1790677960083252 us; error 51 us; skew 500 ppm
Sep 29 10:32:40 dist-test-slave-9cj5 krb5kdc[9629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:40.086076 9642 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:32:40.087095 9642 webserver.cc:460] Webserver started at http://127.0.0.1:40401/ using document root <none> and password file <none>
I20260929 10:32:40.087687 9642 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:40.087778 9642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:40.088016 9642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:40.090523 9642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "dbd5f54353e349de9398624883826172"
format_stamp: "Formatted at 2026-09-29 10:32:40 on dist-test-slave-9cj5"
server_key: "00b75c1812aaa0e40ae5fdbd6815ae8a"
server_key_iv: "e1a6deca04651fae4e62fa721287092f"
server_key_version: "encryptionkey@0"
I20260929 10:32:40.091107 9642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "dbd5f54353e349de9398624883826172"
format_stamp: "Formatted at 2026-09-29 10:32:40 on dist-test-slave-9cj5"
server_key: "00b75c1812aaa0e40ae5fdbd6815ae8a"
server_key_iv: "e1a6deca04651fae4e62fa721287092f"
server_key_version: "encryptionkey@0"
I20260929 10:32:40.094877 9642 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260929 10:32:40.097311 9657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:40.098695 9642 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:32:40.098860 9642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "dbd5f54353e349de9398624883826172"
format_stamp: "Formatted at 2026-09-29 10:32:40 on dist-test-slave-9cj5"
server_key: "00b75c1812aaa0e40ae5fdbd6815ae8a"
server_key_iv: "e1a6deca04651fae4e62fa721287092f"
server_key_version: "encryptionkey@0"
I20260929 10:32:40.098989 9642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:40.128830 9642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:40.131930 9642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:40.132161 9642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:40.139433 9642 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36685
I20260929 10:32:40.139454 9709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36685 every 8 connection(s)
I20260929 10:32:40.140403 9642 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:32:40.143067 9710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:40.145344 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 9642
I20260929 10:32:40.145496 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:32:40.145911 20848 external_mini_cluster.cc:1428] Setting key 2a9d763238808ace20cfd797423f84a0
I20260929 10:32:40.149024 9710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbd5f54353e349de9398624883826172: Bootstrap starting.
I20260929 10:32:40.151415 9710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbd5f54353e349de9398624883826172: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:40.152244 9710 log.cc:824] T 00000000000000000000000000000000 P dbd5f54353e349de9398624883826172: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:40.154670 9710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbd5f54353e349de9398624883826172: No bootstrap required, opened a new log
Sep 29 10:32:40 dist-test-slave-9cj5 krb5kdc[9629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 10:32:40.158548 9710 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbd5f54353e349de9398624883826172 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd5f54353e349de9398624883826172" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36685 } }
I20260929 10:32:40.158849 9710 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbd5f54353e349de9398624883826172 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:40.158923 9710 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbd5f54353e349de9398624883826172 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbd5f54353e349de9398624883826172, State: Initialized, Role: FOLLOWER
I20260929 10:32:40.159389 9710 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbd5f54353e349de9398624883826172 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd5f54353e349de9398624883826172" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36685 } }
I20260929 10:32:40.159508 9710 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbd5f54353e349de9398624883826172 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:32:40.159610 9710 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbd5f54353e349de9398624883826172 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:32:40.159722 9710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbd5f54353e349de9398624883826172 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:40.160665 9710 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbd5f54353e349de9398624883826172 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd5f54353e349de9398624883826172" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36685 } }
I20260929 10:32:40.161082 9710 leader_election.cc:302] T 00000000000000000000000000000000 P dbd5f54353e349de9398624883826172 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbd5f54353e349de9398624883826172; no voters:
I20260929 10:32:40.161417 9710 leader_election.cc:288] T 00000000000000000000000000000000 P dbd5f54353e349de9398624883826172 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:32:40.161544 9715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbd5f54353e349de9398624883826172 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:32:40.161772 9715 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbd5f54353e349de9398624883826172 [term 1 LEADER]: Becoming Leader. State: Replica: dbd5f54353e349de9398624883826172, State: Running, Role: LEADER
I20260929 10:32:40.161952 9713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:40.147356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:40575 (local address 127.0.0.1:36685)
0929 10:32:40.147816 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:40.147830 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:40.147865 (+ 35us) server_negotiation.cc:407] Connection header received
0929 10:32:40.148480 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:40.148497 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:40.148829 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:40.149177 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:40.150067 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:40.151047 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:40.152026 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:40.152552 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:40.152741 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:40.155611 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:40.155617 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:40.155630 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:40.155658 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:40.158069 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:40.158750 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:40.158755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:40.158760 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:40.158950 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:40.159426 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:40.159430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:40.159431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:40.160013 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:40.160264 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:40.160632 (+ 368us) server_negotiation.cc:299] Negotiation successful
0929 10:32:40.160875 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":147,"threads_started":1}
I20260929 10:32:40.162147 9715 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbd5f54353e349de9398624883826172 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd5f54353e349de9398624883826172" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36685 } }
I20260929 10:32:40.162779 9710 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbd5f54353e349de9398624883826172 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:32:40.163348 9715 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbd5f54353e349de9398624883826172 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbd5f54353e349de9398624883826172. Latest consensus state: current_term: 1 leader_uuid: "dbd5f54353e349de9398624883826172" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd5f54353e349de9398624883826172" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36685 } } }
I20260929 10:32:40.163470 9715 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbd5f54353e349de9398624883826172 [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:40.163816 9720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:32:40.163789 9716 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbd5f54353e349de9398624883826172 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbd5f54353e349de9398624883826172" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd5f54353e349de9398624883826172" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36685 } } }
I20260929 10:32:40.164894 9716 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbd5f54353e349de9398624883826172 [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:40.165841 20848 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:32:40.168915 9720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:32:40.175550 9720 catalog_manager.cc:1347] Generated new cluster ID: f93b2a12ff1e42db9ab53cf35a124424
I20260929 10:32:40.175673 9720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:32:40.177071 9713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:40.168015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:50473 (local address 127.0.0.1:36685)
0929 10:32:40.168123 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:40.168128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:40.168271 (+ 143us) server_negotiation.cc:407] Connection header received
0929 10:32:40.168449 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:40.168452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:40.168499 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:40.168561 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:40.169568 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:40.170340 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:40.171357 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:40.171625 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:40.171743 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:40.173285 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:40.173289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:40.173292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:40.173317 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:40.174943 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:40.175578 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:40.175581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:40.175583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:40.175675 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:40.176089 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:40.176093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:40.176095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:40.176332 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:40.176487 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:40.176804 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 10:32:40.176911 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 10:32:40.198765 9720 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:32:40.199784 9720 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:32:40.208114 9720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbd5f54353e349de9398624883826172: Generated new TSK 0
I20260929 10:32:40.208977 9720 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:32:40.213068 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 9642
W20260929 10:32:40.218695 9627 connection.cc:535] client connection to 127.0.0.1:36685 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36685): Connection reset by peer (error 104)
2026-09-29T10:32:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1454 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:32:40 dist-test-slave-9cj5 krb5kdc[9741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:32:40 dist-test-slave-9cj5 krb5kdc[9741](info): set up 2 sockets
Sep 29 10:32:40 dist-test-slave-9cj5 krb5kdc[9741](info): commencing operation
krb5kdc: starting...
W20260929 10:32:41.259452 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:32:41 dist-test-slave-9cj5 krb5kdc[9741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:32:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:32:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:32:41.386101 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40933
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:32:41.496501 9754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:41.496776 9754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:41.496819 9754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:41.500810 9754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:32:41.500871 9754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:41.500890 9754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:41.500907 9754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:32:41.500922 9754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:32:41.506212 9754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40933
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:41.507311 9754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:41.508299 9754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:32:41.516930 9754 server_base.cc:1034] running on GCE node
W20260929 10:32:41.516885 9760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:41.516896 9762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:41.517098 9759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:41.517716 9754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:41.518945 9754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:41.520143 9754 hybrid_clock.cc:648] HybridClock initialized: now 1790677961520129 us; error 30 us; skew 500 ppm
Sep 29 10:32:41 dist-test-slave-9cj5 krb5kdc[9741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:41.527153 9754 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:32:41.528200 9754 webserver.cc:460] Webserver started at http://127.0.0.1:44443/ using document root <none> and password file <none>
I20260929 10:32:41.528746 9754 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:41.528813 9754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:41.529000 9754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:41.531121 9754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "d055c7e6f89a48b7a7396db709d58653"
format_stamp: "Formatted at 2026-09-29 10:32:41 on dist-test-slave-9cj5"
server_key: "976e37b825fc9c0314ac92130cba2fa5"
server_key_iv: "64fe906f625bd517700857965708b8af"
server_key_version: "encryptionkey@0"
I20260929 10:32:41.531711 9754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "d055c7e6f89a48b7a7396db709d58653"
format_stamp: "Formatted at 2026-09-29 10:32:41 on dist-test-slave-9cj5"
server_key: "976e37b825fc9c0314ac92130cba2fa5"
server_key_iv: "64fe906f625bd517700857965708b8af"
server_key_version: "encryptionkey@0"
I20260929 10:32:41.535676 9754 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 10:32:41.538123 9769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:41.539398 9754 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 10:32:41.539496 9754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "d055c7e6f89a48b7a7396db709d58653"
format_stamp: "Formatted at 2026-09-29 10:32:41 on dist-test-slave-9cj5"
server_key: "976e37b825fc9c0314ac92130cba2fa5"
server_key_iv: "64fe906f625bd517700857965708b8af"
server_key_version: "encryptionkey@0"
I20260929 10:32:41.539582 9754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:41.550251 9754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:41.553414 9754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:41.553578 9754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:41.560806 9754 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40933
I20260929 10:32:41.560825 9821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40933 every 8 connection(s)
I20260929 10:32:41.561865 9754 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:32:41.562361 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 9754
I20260929 10:32:41.562531 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:32:41.562825 20848 external_mini_cluster.cc:1428] Setting key bd441d920fd6b6293e86b8392690058f
I20260929 10:32:41.565064 9822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:41.571563 9822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d055c7e6f89a48b7a7396db709d58653: Bootstrap starting.
I20260929 10:32:41.572021 9825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:41.564181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:52177 (local address 127.0.0.1:40933)
0929 10:32:41.564591 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:41.564610 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:41.564655 (+ 45us) server_negotiation.cc:407] Connection header received
0929 10:32:41.565364 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:41.565383 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:41.565722 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:41.566071 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:41.566383 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:41.567366 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:41.568614 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:41.569082 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:41.569213 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:41.570235 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:41.570239 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:41.570252 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:41.570642 (+ 390us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:41.570696 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:41.570719 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:41.570821 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:41.571111 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 10:32:41.571257 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":142,"threads_started":1}
I20260929 10:32:41.573837 9822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d055c7e6f89a48b7a7396db709d58653: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:41.574679 9822 log.cc:824] T 00000000000000000000000000000000 P d055c7e6f89a48b7a7396db709d58653: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:41.574994 20848 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:32:41.576977 9822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d055c7e6f89a48b7a7396db709d58653: No bootstrap required, opened a new log
I20260929 10:32:41.581490 9825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:41.577593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:60087 (local address 127.0.0.1:40933)
0929 10:32:41.577708 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:41.577711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:41.577723 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:32:41.577805 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:41.577810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:41.577855 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:41.577933 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:41.578208 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:41.578696 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:41.579882 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:41.580092 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:41.580170 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:41.581003 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:41.581005 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:41.581007 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:41.581018 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:41.581040 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:41.581049 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:41.581125 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:41.581289 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 10:32:41.581342 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 10:32:41.581714 9822 raft_consensus.cc:348] T 00000000000000000000000000000000 P d055c7e6f89a48b7a7396db709d58653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d055c7e6f89a48b7a7396db709d58653" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40933 } }
I20260929 10:32:41.582031 9822 raft_consensus.cc:374] T 00000000000000000000000000000000 P d055c7e6f89a48b7a7396db709d58653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:41.582094 9822 raft_consensus.cc:729] T 00000000000000000000000000000000 P d055c7e6f89a48b7a7396db709d58653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d055c7e6f89a48b7a7396db709d58653, State: Initialized, Role: FOLLOWER
I20260929 10:32:41.582551 9822 consensus_queue.cc:260] T 00000000000000000000000000000000 P d055c7e6f89a48b7a7396db709d58653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d055c7e6f89a48b7a7396db709d58653" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40933 } }
I20260929 10:32:41.582707 9822 raft_consensus.cc:388] T 00000000000000000000000000000000 P d055c7e6f89a48b7a7396db709d58653 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:32:41.582772 9822 raft_consensus.cc:482] T 00000000000000000000000000000000 P d055c7e6f89a48b7a7396db709d58653 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:32:41.582867 9822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d055c7e6f89a48b7a7396db709d58653 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:41.583779 9822 raft_consensus.cc:504] T 00000000000000000000000000000000 P d055c7e6f89a48b7a7396db709d58653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d055c7e6f89a48b7a7396db709d58653" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40933 } }
I20260929 10:32:41.584121 9822 leader_election.cc:302] T 00000000000000000000000000000000 P d055c7e6f89a48b7a7396db709d58653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d055c7e6f89a48b7a7396db709d58653; no voters:
I20260929 10:32:41.584420 9822 leader_election.cc:288] T 00000000000000000000000000000000 P d055c7e6f89a48b7a7396db709d58653 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:32:41.584517 9832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d055c7e6f89a48b7a7396db709d58653 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:32:41.584782 9832 raft_consensus.cc:686] T 00000000000000000000000000000000 P d055c7e6f89a48b7a7396db709d58653 [term 1 LEADER]: Becoming Leader. State: Replica: d055c7e6f89a48b7a7396db709d58653, State: Running, Role: LEADER
I20260929 10:32:41.585079 9832 consensus_queue.cc:237] T 00000000000000000000000000000000 P d055c7e6f89a48b7a7396db709d58653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d055c7e6f89a48b7a7396db709d58653" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40933 } }
I20260929 10:32:41.585671 9822 sys_catalog.cc:564] T 00000000000000000000000000000000 P d055c7e6f89a48b7a7396db709d58653 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:32:41.586222 9834 sys_catalog.cc:455] T 00000000000000000000000000000000 P d055c7e6f89a48b7a7396db709d58653 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d055c7e6f89a48b7a7396db709d58653. Latest consensus state: current_term: 1 leader_uuid: "d055c7e6f89a48b7a7396db709d58653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d055c7e6f89a48b7a7396db709d58653" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40933 } } }
I20260929 10:32:41.586428 9834 sys_catalog.cc:458] T 00000000000000000000000000000000 P d055c7e6f89a48b7a7396db709d58653 [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:41.586711 9833 sys_catalog.cc:455] T 00000000000000000000000000000000 P d055c7e6f89a48b7a7396db709d58653 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d055c7e6f89a48b7a7396db709d58653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d055c7e6f89a48b7a7396db709d58653" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40933 } } }
I20260929 10:32:41.586817 9833 sys_catalog.cc:458] T 00000000000000000000000000000000 P d055c7e6f89a48b7a7396db709d58653 [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:41.586901 9840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:32:41.590222 9840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:32:41.596163 9840 catalog_manager.cc:1347] Generated new cluster ID: d42d650a173a49abb4448b784e8eaa88
I20260929 10:32:41.596254 9840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:32:41.610719 9840 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:32:41.611706 9840 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:32:41.617733 9840 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d055c7e6f89a48b7a7396db709d58653: Generated new TSK 0
I20260929 10:32:41.618468 9840 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:32:41.628839 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 9754
W20260929 10:32:41.633594 9739 connection.cc:535] client connection to 127.0.0.1:40933 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40933): Connection reset by peer (error 104)
2026-09-29T10:32:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:32:41 dist-test-slave-9cj5 krb5kdc[9853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:32:41 dist-test-slave-9cj5 krb5kdc[9853](info): set up 2 sockets
Sep 29 10:32:41 dist-test-slave-9cj5 krb5kdc[9853](info): commencing operation
krb5kdc: starting...
W20260929 10:32:42.714694 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:32:42 dist-test-slave-9cj5 krb5kdc[9853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:32:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:32:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:32:42.841919 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38483
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:32:42.957041 9866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:42.957365 9866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:42.957458 9866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:42.961663 9866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:32:42.961798 9866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:42.961848 9866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:42.961891 9866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:32:42.961946 9866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:32:42.967368 9866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38483
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677679287822-20848-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:42.968461 9866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:42.969460 9866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:42.977669 9874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:42.977669 9871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:42.977862 9866 server_base.cc:1034] running on GCE node
W20260929 10:32:42.977725 9872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:42.978478 9866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:42.979568 9866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:42.980767 9866 hybrid_clock.cc:648] HybridClock initialized: now 1790677962980740 us; error 38 us; skew 500 ppm
Sep 29 10:32:42 dist-test-slave-9cj5 krb5kdc[9853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:42.983523 9866 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:32:42.984432 9866 webserver.cc:460] Webserver started at http://127.0.0.1:42335/ using document root <none> and password file <none>
I20260929 10:32:42.984992 9866 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:42.985080 9866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:42.985311 9866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:42.987371 9866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "6021515ae14a46c591bbff6d338c9f0a"
format_stamp: "Formatted at 2026-09-29 10:32:42 on dist-test-slave-9cj5"
server_key: "80954a91d25afa34caaa9d5e66c3422b"
server_key_iv: "2c0f9030c03cb5896ee2b1d7e468ced0"
server_key_version: "encryptionkey@0"
I20260929 10:32:42.987941 9866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "6021515ae14a46c591bbff6d338c9f0a"
format_stamp: "Formatted at 2026-09-29 10:32:42 on dist-test-slave-9cj5"
server_key: "80954a91d25afa34caaa9d5e66c3422b"
server_key_iv: "2c0f9030c03cb5896ee2b1d7e468ced0"
server_key_version: "encryptionkey@0"
I20260929 10:32:42.991791 9866 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:32:42.994374 9881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:42.995913 9866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:32:42.996104 9866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "6021515ae14a46c591bbff6d338c9f0a"
format_stamp: "Formatted at 2026-09-29 10:32:42 on dist-test-slave-9cj5"
server_key: "80954a91d25afa34caaa9d5e66c3422b"
server_key_iv: "2c0f9030c03cb5896ee2b1d7e468ced0"
server_key_version: "encryptionkey@0"
I20260929 10:32:42.996243 9866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:43.001477 9866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:43.004928 9866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:43.005149 9866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:43.012377 9866 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38483
I20260929 10:32:43.012403 9933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38483 every 8 connection(s)
I20260929 10:32:43.013307 9866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:32:43.016001 9934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:43.018373 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 9866
I20260929 10:32:43.018512 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:32:43.018827 20848 external_mini_cluster.cc:1428] Setting key aabf60bbf870d01ee080b7744ce96801
I20260929 10:32:43.022156 9934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6021515ae14a46c591bbff6d338c9f0a: Bootstrap starting.
I20260929 10:32:43.024020 9937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:43.020338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:35819 (local address 127.0.0.1:38483)
0929 10:32:43.020787 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:43.020802 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:43.020836 (+ 34us) server_negotiation.cc:407] Connection header received
0929 10:32:43.021466 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:43.021482 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:43.021816 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:43.022151 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:43.022897 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:43.022901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:43.022912 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:43.022922 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:43.022994 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:43.023014 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:43.023160 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:43.023229 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 10:32:43.023379 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":148,"threads_started":1}
I20260929 10:32:43.024575 9934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6021515ae14a46c591bbff6d338c9f0a: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:43.025352 9934 log.cc:824] T 00000000000000000000000000000000 P 6021515ae14a46c591bbff6d338c9f0a: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:43.026683 20848 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:32:43.027385 9934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6021515ae14a46c591bbff6d338c9f0a: No bootstrap required, opened a new log
I20260929 10:32:43.030941 9937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:43.029046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:43539 (local address 127.0.0.1:38483)
0929 10:32:43.029181 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:43.029186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:43.029334 (+ 148us) server_negotiation.cc:407] Connection header received
0929 10:32:43.029434 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:43.029437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:43.029470 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:43.029557 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:43.030377 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:43.030382 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:43.030387 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:43.030403 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:43.030437 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:43.030454 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:43.030569 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:43.030729 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 10:32:43.030781 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 10:32:43.031910 9934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6021515ae14a46c591bbff6d338c9f0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6021515ae14a46c591bbff6d338c9f0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38483 } }
I20260929 10:32:43.032208 9934 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6021515ae14a46c591bbff6d338c9f0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:43.032291 9934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6021515ae14a46c591bbff6d338c9f0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6021515ae14a46c591bbff6d338c9f0a, State: Initialized, Role: FOLLOWER
I20260929 10:32:43.032724 9934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6021515ae14a46c591bbff6d338c9f0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6021515ae14a46c591bbff6d338c9f0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38483 } }
I20260929 10:32:43.032845 9934 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6021515ae14a46c591bbff6d338c9f0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:32:43.032933 9934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6021515ae14a46c591bbff6d338c9f0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:32:43.033030 9934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6021515ae14a46c591bbff6d338c9f0a [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:43.034078 9934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6021515ae14a46c591bbff6d338c9f0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6021515ae14a46c591bbff6d338c9f0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38483 } }
I20260929 10:32:43.034436 9934 leader_election.cc:302] T 00000000000000000000000000000000 P 6021515ae14a46c591bbff6d338c9f0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6021515ae14a46c591bbff6d338c9f0a; no voters:
I20260929 10:32:43.034710 9934 leader_election.cc:288] T 00000000000000000000000000000000 P 6021515ae14a46c591bbff6d338c9f0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:32:43.034842 9944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6021515ae14a46c591bbff6d338c9f0a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:32:43.035091 9944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6021515ae14a46c591bbff6d338c9f0a [term 1 LEADER]: Becoming Leader. State: Replica: 6021515ae14a46c591bbff6d338c9f0a, State: Running, Role: LEADER
I20260929 10:32:43.035458 9944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6021515ae14a46c591bbff6d338c9f0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6021515ae14a46c591bbff6d338c9f0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38483 } }
I20260929 10:32:43.036111 9934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6021515ae14a46c591bbff6d338c9f0a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:32:43.037400 9946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6021515ae14a46c591bbff6d338c9f0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6021515ae14a46c591bbff6d338c9f0a. Latest consensus state: current_term: 1 leader_uuid: "6021515ae14a46c591bbff6d338c9f0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6021515ae14a46c591bbff6d338c9f0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38483 } } }
I20260929 10:32:43.037407 9945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6021515ae14a46c591bbff6d338c9f0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6021515ae14a46c591bbff6d338c9f0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6021515ae14a46c591bbff6d338c9f0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38483 } } }
I20260929 10:32:43.037581 9946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6021515ae14a46c591bbff6d338c9f0a [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:43.037602 9945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6021515ae14a46c591bbff6d338c9f0a [sys.catalog]: This master's current role is: LEADER
W20260929 10:32:43.040787 9959 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6021515ae14a46c591bbff6d338c9f0a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:32:43.040858 9959 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:32:43.040951 9953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:32:43.041806 9953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:32:43.047763 9953 catalog_manager.cc:1347] Generated new cluster ID: 95fe8acf7a1f42dd9503e32a07e5c784
I20260929 10:32:43.047843 9953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:32:43.061245 9953 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:32:43.061653 9953 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:32:43.070892 9953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6021515ae14a46c591bbff6d338c9f0a: Generated new TSK 0
I20260929 10:32:43.071784 9953 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:32:43.081384 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 9866
2026-09-29T10:32:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1453 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:32:43 dist-test-slave-9cj5 krb5kdc[9965](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:32:43 dist-test-slave-9cj5 krb5kdc[9965](info): set up 2 sockets
Sep 29 10:32:43 dist-test-slave-9cj5 krb5kdc[9965](info): commencing operation
krb5kdc: starting...
W20260929 10:32:44.150849 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:32:44 dist-test-slave-9cj5 krb5kdc[9965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:32:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:32:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
I20260929 10:32:44.277673 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:46137
--webserver_interface=127.20.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:43327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:46137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:32:44.385842 9978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:44.386147 9978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:44.386222 9978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:44.390249 9978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:32:44.390345 9978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:44.390385 9978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:44.390424 9978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:32:44.390460 9978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:32:44.395682 9978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:43327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:46137
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:46137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:44.396775 9978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:44.397814 9978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:32:44.406400 9978 server_base.cc:1034] running on GCE node
W20260929 10:32:44.406333 9984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:44.406327 9983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:44.406333 9986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:44.407196 9978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:44.408305 9978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:44.409476 9978 hybrid_clock.cc:648] HybridClock initialized: now 1790677964409458 us; error 31 us; skew 500 ppm
Sep 29 10:32:44 dist-test-slave-9cj5 krb5kdc[9965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677964, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:44.412199 9978 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:32:44.413087 9978 webserver.cc:460] Webserver started at http://127.20.92.62:42787/ using document root <none> and password file <none>
I20260929 10:32:44.413667 9978 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:44.413781 9978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:44.414009 9978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:44.416054 9978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "1e4c7584fa3243e2b8ebc1d5f8d2a337"
format_stamp: "Formatted at 2026-09-29 10:32:44 on dist-test-slave-9cj5"
server_key: "70c26a190ea0e7d9f2e049831c41c4b2"
server_key_iv: "c86b56c1b0fc8a2ec453079325763ce7"
server_key_version: "encryptionkey@0"
I20260929 10:32:44.416625 9978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "1e4c7584fa3243e2b8ebc1d5f8d2a337"
format_stamp: "Formatted at 2026-09-29 10:32:44 on dist-test-slave-9cj5"
server_key: "70c26a190ea0e7d9f2e049831c41c4b2"
server_key_iv: "c86b56c1b0fc8a2ec453079325763ce7"
server_key_version: "encryptionkey@0"
I20260929 10:32:44.420434 9978 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 10:32:44.422938 9993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:44.424321 9978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:32:44.424460 9978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "1e4c7584fa3243e2b8ebc1d5f8d2a337"
format_stamp: "Formatted at 2026-09-29 10:32:44 on dist-test-slave-9cj5"
server_key: "70c26a190ea0e7d9f2e049831c41c4b2"
server_key_iv: "c86b56c1b0fc8a2ec453079325763ce7"
server_key_version: "encryptionkey@0"
I20260929 10:32:44.424590 9978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:44.447521 9978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:44.450742 9978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:44.450947 9978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:44.458186 9978 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:46137
I20260929 10:32:44.458204 10045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:46137 every 8 connection(s)
I20260929 10:32:44.459168 9978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:32:44.462004 10046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:44.464123 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 9978
I20260929 10:32:44.464295 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:32:44.464560 20848 external_mini_cluster.cc:1428] Setting key 5ae84033248acdf3d8ca63a9366bee98
I20260929 10:32:44.467602 10046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e4c7584fa3243e2b8ebc1d5f8d2a337: Bootstrap starting.
I20260929 10:32:44.470139 10046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e4c7584fa3243e2b8ebc1d5f8d2a337: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:44.470975 10046 log.cc:824] T 00000000000000000000000000000000 P 1e4c7584fa3243e2b8ebc1d5f8d2a337: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:44.472965 10046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e4c7584fa3243e2b8ebc1d5f8d2a337: No bootstrap required, opened a new log
Sep 29 10:32:44 dist-test-slave-9cj5 krb5kdc[9965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:32:44.477345 10046 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e4c7584fa3243e2b8ebc1d5f8d2a337 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4c7584fa3243e2b8ebc1d5f8d2a337" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 46137 } }
I20260929 10:32:44.477676 10046 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e4c7584fa3243e2b8ebc1d5f8d2a337 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:44.477730 10046 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e4c7584fa3243e2b8ebc1d5f8d2a337 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e4c7584fa3243e2b8ebc1d5f8d2a337, State: Initialized, Role: FOLLOWER
I20260929 10:32:44.478246 10046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e4c7584fa3243e2b8ebc1d5f8d2a337 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4c7584fa3243e2b8ebc1d5f8d2a337" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 46137 } }
I20260929 10:32:44.478418 10046 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e4c7584fa3243e2b8ebc1d5f8d2a337 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:32:44.478510 10046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e4c7584fa3243e2b8ebc1d5f8d2a337 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:32:44.478611 10046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e4c7584fa3243e2b8ebc1d5f8d2a337 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:44.479799 10046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e4c7584fa3243e2b8ebc1d5f8d2a337 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4c7584fa3243e2b8ebc1d5f8d2a337" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 46137 } }
I20260929 10:32:44.480039 10049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:44.465898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39085 (local address 127.20.92.62:46137)
0929 10:32:44.466296 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:44.466311 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:44.466345 (+ 34us) server_negotiation.cc:407] Connection header received
0929 10:32:44.466971 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:44.466986 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:44.467332 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:44.467611 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:44.468488 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:44.469482 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:44.470676 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:44.471222 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:44.471367 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:44.474030 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:44.474036 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:44.474053 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:44.474084 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:44.476556 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:44.477130 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:44.477135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:44.477141 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:44.477281 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:44.477632 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:44.477635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:44.477639 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:44.478183 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:44.478416 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:44.478809 (+ 393us) server_negotiation.cc:299] Negotiation successful
0929 10:32:44.479046 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":134,"threads_started":1}
I20260929 10:32:44.480182 10046 leader_election.cc:302] T 00000000000000000000000000000000 P 1e4c7584fa3243e2b8ebc1d5f8d2a337 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e4c7584fa3243e2b8ebc1d5f8d2a337; no voters:
I20260929 10:32:44.480566 10046 leader_election.cc:288] T 00000000000000000000000000000000 P 1e4c7584fa3243e2b8ebc1d5f8d2a337 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:32:44.480690 10051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e4c7584fa3243e2b8ebc1d5f8d2a337 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:32:44.480902 10051 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e4c7584fa3243e2b8ebc1d5f8d2a337 [term 1 LEADER]: Becoming Leader. State: Replica: 1e4c7584fa3243e2b8ebc1d5f8d2a337, State: Running, Role: LEADER
I20260929 10:32:44.481266 10051 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e4c7584fa3243e2b8ebc1d5f8d2a337 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4c7584fa3243e2b8ebc1d5f8d2a337" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 46137 } }
I20260929 10:32:44.481931 10046 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e4c7584fa3243e2b8ebc1d5f8d2a337 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:32:44.482973 10053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e4c7584fa3243e2b8ebc1d5f8d2a337 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e4c7584fa3243e2b8ebc1d5f8d2a337. Latest consensus state: current_term: 1 leader_uuid: "1e4c7584fa3243e2b8ebc1d5f8d2a337" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4c7584fa3243e2b8ebc1d5f8d2a337" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 46137 } } }
I20260929 10:32:44.483098 10053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e4c7584fa3243e2b8ebc1d5f8d2a337 [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:44.483312 20848 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:32:44.483414 10052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e4c7584fa3243e2b8ebc1d5f8d2a337 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e4c7584fa3243e2b8ebc1d5f8d2a337" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4c7584fa3243e2b8ebc1d5f8d2a337" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 46137 } } }
I20260929 10:32:44.483505 10052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e4c7584fa3243e2b8ebc1d5f8d2a337 [sys.catalog]: This master's current role is: LEADER
W20260929 10:32:44.486881 10066 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e4c7584fa3243e2b8ebc1d5f8d2a337: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:32:44.486963 10066 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:32:44.487031 10060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:32:44.487902 10060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:32:44.494174 10060 catalog_manager.cc:1347] Generated new cluster ID: 826086b5cfc94dc7888167448b207407
I20260929 10:32:44.494249 10060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:32:44.494175 10049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:44.485435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54813 (local address 127.20.92.62:46137)
0929 10:32:44.485573 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:44.485577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:44.486223 (+ 646us) server_negotiation.cc:407] Connection header received
0929 10:32:44.486383 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:44.486386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:44.486430 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:44.486519 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:44.487271 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:44.487792 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:44.488635 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:44.488883 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:44.488994 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:44.490560 (+ 1566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:44.490564 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:44.490566 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:44.490590 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:44.492135 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:44.492683 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:44.492686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:44.492688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:44.492770 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:44.493150 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:44.493153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:44.493155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:44.493360 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:44.493473 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:44.493931 (+ 458us) server_negotiation.cc:299] Negotiation successful
0929 10:32:44.494023 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 10:32:44.511061 10060 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:32:44.512110 10060 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:32:44.520905 10060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e4c7584fa3243e2b8ebc1d5f8d2a337: Generated new TSK 0
I20260929 10:32:44.521829 10060 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:32:44.532594 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 9978
W20260929 10:32:44.538020 9963 connection.cc:535] client connection to 127.20.92.62:46137 recv error: Network error: failed to read from TLS socket (remote: 127.20.92.62:46137): Connection reset by peer (error 104)
2026-09-29T10:32:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1451 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:32:44 dist-test-slave-9cj5 krb5kdc[10077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:32:44 dist-test-slave-9cj5 krb5kdc[10077](info): set up 2 sockets
Sep 29 10:32:44 dist-test-slave-9cj5 krb5kdc[10077](info): commencing operation
krb5kdc: starting...
W20260929 10:32:45.642849 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:32:45 dist-test-slave-9cj5 krb5kdc[10077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:32:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:32:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
I20260929 10:32:45.769133 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:35797
--webserver_interface=127.20.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:35797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:32:45.880210 10090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:45.880538 10090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:45.880627 10090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:45.884812 10090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:32:45.884900 10090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:45.884938 10090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:45.884996 10090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:32:45.885035 10090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:32:45.890417 10090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:35797
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:35797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:45.891508 10090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:45.892552 10090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:45.901099 10098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:45.901134 10096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:45.901268 10090 server_base.cc:1034] running on GCE node
W20260929 10:32:45.901093 10095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:45.902052 10090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:45.903276 10090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:45.904469 10090 hybrid_clock.cc:648] HybridClock initialized: now 1790677965904452 us; error 33 us; skew 500 ppm
Sep 29 10:32:45 dist-test-slave-9cj5 krb5kdc[10077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677965, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:45.907356 10090 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:32:45.908211 10090 webserver.cc:460] Webserver started at http://127.20.92.62:33721/ using document root <none> and password file <none>
I20260929 10:32:45.908788 10090 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:45.908850 10090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:45.909014 10090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:45.911078 10090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "4a68822baa704b02b4dd661754ded0dc"
format_stamp: "Formatted at 2026-09-29 10:32:45 on dist-test-slave-9cj5"
server_key: "24c93fe880ffa228d01b48a56990dc25"
server_key_iv: "3951e86f8b0751e17dd0813010b30e1a"
server_key_version: "encryptionkey@0"
I20260929 10:32:45.911585 10090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "4a68822baa704b02b4dd661754ded0dc"
format_stamp: "Formatted at 2026-09-29 10:32:45 on dist-test-slave-9cj5"
server_key: "24c93fe880ffa228d01b48a56990dc25"
server_key_iv: "3951e86f8b0751e17dd0813010b30e1a"
server_key_version: "encryptionkey@0"
I20260929 10:32:45.915388 10090 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:32:45.917721 10105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:45.918967 10090 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:32:45.919104 10090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "4a68822baa704b02b4dd661754ded0dc"
format_stamp: "Formatted at 2026-09-29 10:32:45 on dist-test-slave-9cj5"
server_key: "24c93fe880ffa228d01b48a56990dc25"
server_key_iv: "3951e86f8b0751e17dd0813010b30e1a"
server_key_version: "encryptionkey@0"
I20260929 10:32:45.919212 10090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:45.932770 10090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:45.936057 10090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:45.936265 10090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:45.943197 10090 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:35797
I20260929 10:32:45.943252 10157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:35797 every 8 connection(s)
I20260929 10:32:45.944154 10090 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:32:45.945264 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 10090
I20260929 10:32:45.945418 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:32:45.945731 20848 external_mini_cluster.cc:1428] Setting key 0ee315c2aad58802fa31628f43baf60f
I20260929 10:32:45.947528 10158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:45.953029 10158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a68822baa704b02b4dd661754ded0dc: Bootstrap starting.
I20260929 10:32:45.954588 10161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:45.947152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35263 (local address 127.20.92.62:35797)
0929 10:32:45.947659 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:45.947684 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:45.947719 (+ 35us) server_negotiation.cc:407] Connection header received
0929 10:32:45.948354 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:45.948370 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:45.948690 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:45.949016 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:45.949411 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:45.950455 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:45.951337 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:45.951816 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:45.951988 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:45.952914 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:45.952918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:45.952930 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:45.953353 (+ 423us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:45.953405 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:45.953428 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:45.953568 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:45.953788 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 10:32:45.953921 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":143,"threads_started":1}
I20260929 10:32:45.955358 10158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a68822baa704b02b4dd661754ded0dc: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:45.956214 10158 log.cc:824] T 00000000000000000000000000000000 P 4a68822baa704b02b4dd661754ded0dc: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:45.957561 20848 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:32:45.958281 10158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a68822baa704b02b4dd661754ded0dc: No bootstrap required, opened a new log
I20260929 10:32:45.962672 10158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a68822baa704b02b4dd661754ded0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a68822baa704b02b4dd661754ded0dc" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 35797 } }
I20260929 10:32:45.962927 10158 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a68822baa704b02b4dd661754ded0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:45.963013 10158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a68822baa704b02b4dd661754ded0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a68822baa704b02b4dd661754ded0dc, State: Initialized, Role: FOLLOWER
I20260929 10:32:45.963501 10158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a68822baa704b02b4dd661754ded0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a68822baa704b02b4dd661754ded0dc" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 35797 } }
I20260929 10:32:45.963668 10158 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a68822baa704b02b4dd661754ded0dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:32:45.963740 10158 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a68822baa704b02b4dd661754ded0dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:32:45.963850 10158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a68822baa704b02b4dd661754ded0dc [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:45.964133 10161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:45.959591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40777 (local address 127.20.92.62:35797)
0929 10:32:45.959710 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:45.959714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:45.959892 (+ 178us) server_negotiation.cc:407] Connection header received
0929 10:32:45.959986 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:45.959989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:45.960036 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:45.960129 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:45.960369 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:45.961231 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:45.962089 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:45.962405 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:45.962508 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:45.963483 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:45.963487 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:45.963490 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:45.963503 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:45.963535 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:45.963546 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:45.963648 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:45.963979 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 10:32:45.964026 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 10:32:45.965065 10158 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a68822baa704b02b4dd661754ded0dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a68822baa704b02b4dd661754ded0dc" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 35797 } }
I20260929 10:32:45.965425 10158 leader_election.cc:302] T 00000000000000000000000000000000 P 4a68822baa704b02b4dd661754ded0dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a68822baa704b02b4dd661754ded0dc; no voters:
I20260929 10:32:45.965700 10158 leader_election.cc:288] T 00000000000000000000000000000000 P 4a68822baa704b02b4dd661754ded0dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:32:45.965828 10168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a68822baa704b02b4dd661754ded0dc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:32:45.966050 10168 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a68822baa704b02b4dd661754ded0dc [term 1 LEADER]: Becoming Leader. State: Replica: 4a68822baa704b02b4dd661754ded0dc, State: Running, Role: LEADER
I20260929 10:32:45.966401 10168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a68822baa704b02b4dd661754ded0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a68822baa704b02b4dd661754ded0dc" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 35797 } }
I20260929 10:32:45.967074 10158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a68822baa704b02b4dd661754ded0dc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:32:45.968304 10170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a68822baa704b02b4dd661754ded0dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a68822baa704b02b4dd661754ded0dc. Latest consensus state: current_term: 1 leader_uuid: "4a68822baa704b02b4dd661754ded0dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a68822baa704b02b4dd661754ded0dc" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 35797 } } }
I20260929 10:32:45.968304 10169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a68822baa704b02b4dd661754ded0dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a68822baa704b02b4dd661754ded0dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a68822baa704b02b4dd661754ded0dc" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 35797 } } }
I20260929 10:32:45.968490 10170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a68822baa704b02b4dd661754ded0dc [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:45.968505 10169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a68822baa704b02b4dd661754ded0dc [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:45.968926 10179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:32:45.971640 10179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:32:45.977201 10179 catalog_manager.cc:1347] Generated new cluster ID: 42c92a6541b84e95b2d38724e01ae3e4
I20260929 10:32:45.977272 10179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:32:46.000775 10179 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:32:46.002157 10179 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:32:46.008042 10179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a68822baa704b02b4dd661754ded0dc: Generated new TSK 0
I20260929 10:32:46.008828 10179 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:32:46.017370 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 10090
W20260929 10:32:46.022935 10075 connection.cc:535] client connection to 127.20.92.62:35797 recv error: Network error: failed to read from TLS socket (remote: 127.20.92.62:35797): Connection reset by peer (error 104)
2026-09-29T10:32:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1485 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:32:46 dist-test-slave-9cj5 krb5kdc[10189](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:32:46 dist-test-slave-9cj5 krb5kdc[10189](info): set up 2 sockets
Sep 29 10:32:46 dist-test-slave-9cj5 krb5kdc[10189](info): commencing operation
krb5kdc: starting...
W20260929 10:32:47.058874 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:32:47 dist-test-slave-9cj5 krb5kdc[10189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:32:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:32:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
I20260929 10:32:47.186244 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:42779
--webserver_interface=127.20.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:43947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:42779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:32:47.296648 10202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:47.296958 10202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:47.297037 10202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:47.301086 10202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:32:47.301208 10202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:47.301258 10202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:47.301316 10202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:32:47.301354 10202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:32:47.306789 10202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:43947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:42779
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:42779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:47.307942 10202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:47.308961 10202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:47.317735 10208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:47.317694 10207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:47.317778 10202 server_base.cc:1034] running on GCE node
W20260929 10:32:47.317700 10210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:47.318555 10202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:47.319701 10202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:47.320886 10202 hybrid_clock.cc:648] HybridClock initialized: now 1790677967320861 us; error 38 us; skew 500 ppm
Sep 29 10:32:47 dist-test-slave-9cj5 krb5kdc[10189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677967, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:47.327622 10202 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:32:47.328651 10202 webserver.cc:460] Webserver started at http://127.20.92.62:34251/ using document root <none> and password file <none>
I20260929 10:32:47.329269 10202 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:47.329371 10202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:47.329634 10202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:47.331934 10202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "5fad865a5a7446329bb9c3133d5c8fcd"
format_stamp: "Formatted at 2026-09-29 10:32:47 on dist-test-slave-9cj5"
server_key: "db059ae7843a676d2f71fd7e8e139f8d"
server_key_iv: "b0f57ac9f5cec1a85eea1109afd546a9"
server_key_version: "encryptionkey@0"
I20260929 10:32:47.332614 10202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "5fad865a5a7446329bb9c3133d5c8fcd"
format_stamp: "Formatted at 2026-09-29 10:32:47 on dist-test-slave-9cj5"
server_key: "db059ae7843a676d2f71fd7e8e139f8d"
server_key_iv: "b0f57ac9f5cec1a85eea1109afd546a9"
server_key_version: "encryptionkey@0"
I20260929 10:32:47.336725 10202 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 10:32:47.339421 10217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:47.340674 10202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:32:47.340826 10202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "5fad865a5a7446329bb9c3133d5c8fcd"
format_stamp: "Formatted at 2026-09-29 10:32:47 on dist-test-slave-9cj5"
server_key: "db059ae7843a676d2f71fd7e8e139f8d"
server_key_iv: "b0f57ac9f5cec1a85eea1109afd546a9"
server_key_version: "encryptionkey@0"
I20260929 10:32:47.340946 10202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:47.345696 10202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:47.349114 10202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:47.349355 10202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:47.356514 10202 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:42779
I20260929 10:32:47.356540 10269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:42779 every 8 connection(s)
I20260929 10:32:47.357457 10202 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:32:47.360270 10270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:47.362694 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 10202
I20260929 10:32:47.362854 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:32:47.363157 20848 external_mini_cluster.cc:1428] Setting key f12fb0cdae104d47055bd754a439b5a7
I20260929 10:32:47.366950 10270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fad865a5a7446329bb9c3133d5c8fcd: Bootstrap starting.
I20260929 10:32:47.369585 10273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:47.364791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59681 (local address 127.20.92.62:42779)
0929 10:32:47.365267 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:47.365286 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:47.365331 (+ 45us) server_negotiation.cc:407] Connection header received
0929 10:32:47.366199 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:47.366223 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:47.366586 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:47.366954 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:47.368088 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:47.368093 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:47.368107 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:47.368118 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:47.368175 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:47.368208 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:47.368362 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:47.368516 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 10:32:47.368672 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":138,"threads_started":1}
I20260929 10:32:47.369908 10270 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fad865a5a7446329bb9c3133d5c8fcd: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:47.370738 10270 log.cc:824] T 00000000000000000000000000000000 P 5fad865a5a7446329bb9c3133d5c8fcd: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:47.372709 20848 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:32:47.373185 10270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fad865a5a7446329bb9c3133d5c8fcd: No bootstrap required, opened a new log
I20260929 10:32:47.376745 10273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:47.375018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34673 (local address 127.20.92.62:42779)
0929 10:32:47.375156 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:47.375161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:47.375369 (+ 208us) server_negotiation.cc:407] Connection header received
0929 10:32:47.375488 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:47.375491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:47.375521 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:47.375599 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:47.376306 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:47.376308 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:47.376310 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:47.376320 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:47.376344 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:47.376354 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:47.376424 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:47.376550 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 10:32:47.376583 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 10:32:47.377985 10270 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fad865a5a7446329bb9c3133d5c8fcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fad865a5a7446329bb9c3133d5c8fcd" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 42779 } }
I20260929 10:32:47.378248 10270 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fad865a5a7446329bb9c3133d5c8fcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:47.378307 10270 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fad865a5a7446329bb9c3133d5c8fcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fad865a5a7446329bb9c3133d5c8fcd, State: Initialized, Role: FOLLOWER
I20260929 10:32:47.378726 10270 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fad865a5a7446329bb9c3133d5c8fcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fad865a5a7446329bb9c3133d5c8fcd" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 42779 } }
I20260929 10:32:47.378882 10270 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fad865a5a7446329bb9c3133d5c8fcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:32:47.378950 10270 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fad865a5a7446329bb9c3133d5c8fcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:32:47.379045 10270 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fad865a5a7446329bb9c3133d5c8fcd [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:47.380134 10270 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fad865a5a7446329bb9c3133d5c8fcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fad865a5a7446329bb9c3133d5c8fcd" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 42779 } }
I20260929 10:32:47.380496 10270 leader_election.cc:302] T 00000000000000000000000000000000 P 5fad865a5a7446329bb9c3133d5c8fcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fad865a5a7446329bb9c3133d5c8fcd; no voters:
I20260929 10:32:47.380867 10270 leader_election.cc:288] T 00000000000000000000000000000000 P 5fad865a5a7446329bb9c3133d5c8fcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:32:47.381022 10280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fad865a5a7446329bb9c3133d5c8fcd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:32:47.381255 10280 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fad865a5a7446329bb9c3133d5c8fcd [term 1 LEADER]: Becoming Leader. State: Replica: 5fad865a5a7446329bb9c3133d5c8fcd, State: Running, Role: LEADER
I20260929 10:32:47.381599 10280 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fad865a5a7446329bb9c3133d5c8fcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fad865a5a7446329bb9c3133d5c8fcd" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 42779 } }
I20260929 10:32:47.382210 10270 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fad865a5a7446329bb9c3133d5c8fcd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:32:47.383476 10281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fad865a5a7446329bb9c3133d5c8fcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fad865a5a7446329bb9c3133d5c8fcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fad865a5a7446329bb9c3133d5c8fcd" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 42779 } } }
I20260929 10:32:47.383618 10281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fad865a5a7446329bb9c3133d5c8fcd [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:47.383437 10282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fad865a5a7446329bb9c3133d5c8fcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fad865a5a7446329bb9c3133d5c8fcd. Latest consensus state: current_term: 1 leader_uuid: "5fad865a5a7446329bb9c3133d5c8fcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fad865a5a7446329bb9c3133d5c8fcd" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 42779 } } }
I20260929 10:32:47.383930 10282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fad865a5a7446329bb9c3133d5c8fcd [sys.catalog]: This master's current role is: LEADER
W20260929 10:32:47.387055 10295 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5fad865a5a7446329bb9c3133d5c8fcd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:32:47.387144 10295 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:32:47.387270 10290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:32:47.388212 10290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:32:47.394027 10290 catalog_manager.cc:1347] Generated new cluster ID: 5ac4c7ba58a84a7480e6e7c4df0d746c
I20260929 10:32:47.394104 10290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:32:47.407541 10290 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:32:47.408010 10290 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:32:47.415313 10290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fad865a5a7446329bb9c3133d5c8fcd: Generated new TSK 0
I20260929 10:32:47.416213 10290 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:32:47.420356 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 10202
2026-09-29T10:32:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:32:47 dist-test-slave-9cj5 krb5kdc[10301](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:32:47 dist-test-slave-9cj5 krb5kdc[10301](info): set up 2 sockets
Sep 29 10:32:47 dist-test-slave-9cj5 krb5kdc[10301](info): commencing operation
krb5kdc: starting...
W20260929 10:32:48.458984 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:32:48 dist-test-slave-9cj5 krb5kdc[10301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:32:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:32:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
I20260929 10:32:48.585114 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:43465
--webserver_interface=127.20.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:43465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:32:48.697252 10314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:48.697564 10314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:48.697650 10314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:48.701905 10314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:32:48.701990 10314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:48.702028 10314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:48.702085 10314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:32:48.702126 10314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:32:48.707506 10314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:43465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:43465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:48.708606 10314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:48.709649 10314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:32:48.718377 10314 server_base.cc:1034] running on GCE node
W20260929 10:32:48.718304 10319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:48.718340 10320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:48.718339 10322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:48.719219 10314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:48.720296 10314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:48.721498 10314 hybrid_clock.cc:648] HybridClock initialized: now 1790677968721443 us; error 41 us; skew 500 ppm
Sep 29 10:32:48 dist-test-slave-9cj5 krb5kdc[10301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677968, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:48.724371 10314 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:32:48.725250 10314 webserver.cc:460] Webserver started at http://127.20.92.62:45545/ using document root <none> and password file <none>
I20260929 10:32:48.725869 10314 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:48.725936 10314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:48.726150 10314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:48.728212 10314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "c4cd8a42327d46f982057bbd1d5d126c"
format_stamp: "Formatted at 2026-09-29 10:32:48 on dist-test-slave-9cj5"
server_key: "c9c31cbcd33f4c2ce5665182efe9f517"
server_key_iv: "2f7e5c19e8d7c6a3b17513e74a64d069"
server_key_version: "encryptionkey@0"
I20260929 10:32:48.728778 10314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "c4cd8a42327d46f982057bbd1d5d126c"
format_stamp: "Formatted at 2026-09-29 10:32:48 on dist-test-slave-9cj5"
server_key: "c9c31cbcd33f4c2ce5665182efe9f517"
server_key_iv: "2f7e5c19e8d7c6a3b17513e74a64d069"
server_key_version: "encryptionkey@0"
I20260929 10:32:48.732540 10314 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 10:32:48.735091 10329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:48.736308 10314 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:32:48.736457 10314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "c4cd8a42327d46f982057bbd1d5d126c"
format_stamp: "Formatted at 2026-09-29 10:32:48 on dist-test-slave-9cj5"
server_key: "c9c31cbcd33f4c2ce5665182efe9f517"
server_key_iv: "2f7e5c19e8d7c6a3b17513e74a64d069"
server_key_version: "encryptionkey@0"
I20260929 10:32:48.736578 10314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:48.760856 10314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:48.764230 10314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:48.764515 10314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:48.772358 10314 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:43465
I20260929 10:32:48.772369 10381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:43465 every 8 connection(s)
I20260929 10:32:48.773363 10314 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:32:48.776062 10382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:48.781327 10382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4cd8a42327d46f982057bbd1d5d126c: Bootstrap starting.
I20260929 10:32:48.782025 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 10314
I20260929 10:32:48.782178 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:32:48.782473 20848 external_mini_cluster.cc:1428] Setting key e3e93696f9156606cf4c7ba8c5c3df3d
I20260929 10:32:48.784137 10382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4cd8a42327d46f982057bbd1d5d126c: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:48.785138 10382 log.cc:824] T 00000000000000000000000000000000 P c4cd8a42327d46f982057bbd1d5d126c: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:48.787349 10382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4cd8a42327d46f982057bbd1d5d126c: No bootstrap required, opened a new log
I20260929 10:32:48.791378 10382 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4cd8a42327d46f982057bbd1d5d126c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4cd8a42327d46f982057bbd1d5d126c" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 43465 } }
I20260929 10:32:48.791658 10382 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4cd8a42327d46f982057bbd1d5d126c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:48.791713 10382 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4cd8a42327d46f982057bbd1d5d126c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4cd8a42327d46f982057bbd1d5d126c, State: Initialized, Role: FOLLOWER
Sep 29 10:32:48 dist-test-slave-9cj5 krb5kdc[10301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:32:48.792167 10382 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4cd8a42327d46f982057bbd1d5d126c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4cd8a42327d46f982057bbd1d5d126c" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 43465 } }
I20260929 10:32:48.792331 10382 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4cd8a42327d46f982057bbd1d5d126c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:32:48.792408 10382 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4cd8a42327d46f982057bbd1d5d126c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:32:48.792541 10382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4cd8a42327d46f982057bbd1d5d126c [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:48.793792 10382 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4cd8a42327d46f982057bbd1d5d126c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4cd8a42327d46f982057bbd1d5d126c" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 43465 } }
I20260929 10:32:48.794158 10382 leader_election.cc:302] T 00000000000000000000000000000000 P c4cd8a42327d46f982057bbd1d5d126c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4cd8a42327d46f982057bbd1d5d126c; no voters:
I20260929 10:32:48.794533 10382 leader_election.cc:288] T 00000000000000000000000000000000 P c4cd8a42327d46f982057bbd1d5d126c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:32:48.794580 10387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4cd8a42327d46f982057bbd1d5d126c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:32:48.794847 10387 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4cd8a42327d46f982057bbd1d5d126c [term 1 LEADER]: Becoming Leader. State: Replica: c4cd8a42327d46f982057bbd1d5d126c, State: Running, Role: LEADER
I20260929 10:32:48.795233 10387 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4cd8a42327d46f982057bbd1d5d126c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4cd8a42327d46f982057bbd1d5d126c" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 43465 } }
I20260929 10:32:48.795823 10382 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4cd8a42327d46f982057bbd1d5d126c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:32:48.796901 10388 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4cd8a42327d46f982057bbd1d5d126c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4cd8a42327d46f982057bbd1d5d126c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4cd8a42327d46f982057bbd1d5d126c" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 43465 } } }
I20260929 10:32:48.797030 10388 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4cd8a42327d46f982057bbd1d5d126c [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:48.797403 10389 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4cd8a42327d46f982057bbd1d5d126c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4cd8a42327d46f982057bbd1d5d126c. Latest consensus state: current_term: 1 leader_uuid: "c4cd8a42327d46f982057bbd1d5d126c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4cd8a42327d46f982057bbd1d5d126c" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 43465 } } }
I20260929 10:32:48.797570 10389 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4cd8a42327d46f982057bbd1d5d126c [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:48.798198 10396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:32:48.798297 10385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:48.783722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51477 (local address 127.20.92.62:43465)
0929 10:32:48.784197 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:48.784216 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:48.784259 (+ 43us) server_negotiation.cc:407] Connection header received
0929 10:32:48.785261 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:48.785284 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:48.785691 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:48.786103 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:48.786753 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:48.788201 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:48.789039 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:48.789735 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:48.789940 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:48.792540 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:48.792546 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:48.792566 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:48.792598 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:48.794556 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:48.795138 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:48.795143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:48.795149 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:48.795288 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:48.795612 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:48.795615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:48.795617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:48.796152 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:48.796323 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:48.797027 (+ 704us) server_negotiation.cc:299] Negotiation successful
0929 10:32:48.797312 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":111,"threads_started":1}
I20260929 10:32:48.801036 20848 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:32:48.801216 10396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:32:48.807348 10396 catalog_manager.cc:1347] Generated new cluster ID: 7533d4db7d1440a3a096ab5aae2024fc
I20260929 10:32:48.807418 10396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:32:48.811375 10385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:48.803023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55435 (local address 127.20.92.62:43465)
0929 10:32:48.803130 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:48.803134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:48.803372 (+ 238us) server_negotiation.cc:407] Connection header received
0929 10:32:48.803484 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:48.803486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:48.803522 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:48.803609 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:48.804377 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:48.804898 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:48.805858 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:48.806090 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:48.806204 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:48.807810 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:48.807815 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:48.807818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:48.807849 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:48.809579 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:48.810079 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:48.810083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:48.810086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:48.810185 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:48.810473 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:48.810476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:48.810478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:48.810706 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:48.810852 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:48.811073 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 10:32:48.811204 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 10:32:48.823565 10396 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:32:48.824728 10396 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:32:48.832528 10396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4cd8a42327d46f982057bbd1d5d126c: Generated new TSK 0
I20260929 10:32:48.833351 10396 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:32:48.842962 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 10314
2026-09-29T10:32:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:32:48 dist-test-slave-9cj5 krb5kdc[10413](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:32:48 dist-test-slave-9cj5 krb5kdc[10413](info): set up 2 sockets
Sep 29 10:32:48 dist-test-slave-9cj5 krb5kdc[10413](info): commencing operation
krb5kdc: starting...
W20260929 10:32:49.926819 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:32:50 dist-test-slave-9cj5 krb5kdc[10413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:32:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:32:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
I20260929 10:32:50.056689 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:44935
--webserver_interface=127.20.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:35075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:44935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:32:50.173416 10426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:50.173715 10426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:50.173830 10426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:50.178177 10426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:32:50.178259 10426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:50.178279 10426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:50.178296 10426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:32:50.178313 10426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:32:50.184090 10426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:35075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:44935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:44935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:50.185213 10426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:50.186240 10426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:50.194981 10432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:50.194968 10434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:50.195158 10426 server_base.cc:1034] running on GCE node
W20260929 10:32:50.194963 10431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:50.195925 10426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:50.197079 10426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:50.198252 10426 hybrid_clock.cc:648] HybridClock initialized: now 1790677970198223 us; error 40 us; skew 500 ppm
Sep 29 10:32:50 dist-test-slave-9cj5 krb5kdc[10413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677970, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:50.201028 10426 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:32:50.202004 10426 webserver.cc:460] Webserver started at http://127.20.92.62:42171/ using document root <none> and password file <none>
I20260929 10:32:50.202558 10426 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:50.202616 10426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:50.202826 10426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:50.204854 10426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "2abed552a2fd4aa195d2033e47b8e64f"
format_stamp: "Formatted at 2026-09-29 10:32:50 on dist-test-slave-9cj5"
server_key: "0341e2d2d502d1fbdce42ed1cf4acc72"
server_key_iv: "c54541b62862c79d2dbd1fa16cf52a51"
server_key_version: "encryptionkey@0"
I20260929 10:32:50.205415 10426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "2abed552a2fd4aa195d2033e47b8e64f"
format_stamp: "Formatted at 2026-09-29 10:32:50 on dist-test-slave-9cj5"
server_key: "0341e2d2d502d1fbdce42ed1cf4acc72"
server_key_iv: "c54541b62862c79d2dbd1fa16cf52a51"
server_key_version: "encryptionkey@0"
I20260929 10:32:50.209179 10426 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260929 10:32:50.211930 10441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:50.213418 10426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:32:50.213574 10426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "2abed552a2fd4aa195d2033e47b8e64f"
format_stamp: "Formatted at 2026-09-29 10:32:50 on dist-test-slave-9cj5"
server_key: "0341e2d2d502d1fbdce42ed1cf4acc72"
server_key_iv: "c54541b62862c79d2dbd1fa16cf52a51"
server_key_version: "encryptionkey@0"
I20260929 10:32:50.213698 10426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:50.242938 10426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:50.246623 10426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:50.246842 10426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:50.254484 10426 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:44935
I20260929 10:32:50.254503 10493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:44935 every 8 connection(s)
I20260929 10:32:50.255458 10426 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:32:50.258246 10494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:50.263744 10494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2abed552a2fd4aa195d2033e47b8e64f: Bootstrap starting.
I20260929 10:32:50.263904 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 10426
I20260929 10:32:50.264065 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:32:50.264366 20848 external_mini_cluster.cc:1428] Setting key 296bc8f8ff28fbd1f6ce04fbe560e658
I20260929 10:32:50.266492 10494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2abed552a2fd4aa195d2033e47b8e64f: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:50.267241 10494 log.cc:824] T 00000000000000000000000000000000 P 2abed552a2fd4aa195d2033e47b8e64f: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:50.269227 10494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2abed552a2fd4aa195d2033e47b8e64f: No bootstrap required, opened a new log
I20260929 10:32:50.273656 10497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:50.265740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59075 (local address 127.20.92.62:44935)
0929 10:32:50.266229 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:50.266245 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:50.266280 (+ 35us) server_negotiation.cc:407] Connection header received
0929 10:32:50.267032 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:50.267051 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:50.267458 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:50.267821 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:50.268229 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:50.269172 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:50.270455 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:50.270944 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:50.271075 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:50.271923 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:50.271928 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:50.271941 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:50.272320 (+ 379us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:50.272375 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:50.272398 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:50.272477 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:50.272631 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 10:32:50.272896 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":142,"threads_started":1}
I20260929 10:32:50.273594 10494 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2abed552a2fd4aa195d2033e47b8e64f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abed552a2fd4aa195d2033e47b8e64f" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 44935 } }
I20260929 10:32:50.273872 10494 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2abed552a2fd4aa195d2033e47b8e64f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:50.273929 10494 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2abed552a2fd4aa195d2033e47b8e64f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2abed552a2fd4aa195d2033e47b8e64f, State: Initialized, Role: FOLLOWER
I20260929 10:32:50.274441 10494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2abed552a2fd4aa195d2033e47b8e64f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abed552a2fd4aa195d2033e47b8e64f" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 44935 } }
I20260929 10:32:50.274736 10494 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2abed552a2fd4aa195d2033e47b8e64f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:32:50.274820 10494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2abed552a2fd4aa195d2033e47b8e64f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:32:50.274909 10494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2abed552a2fd4aa195d2033e47b8e64f [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:50.276089 10494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2abed552a2fd4aa195d2033e47b8e64f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abed552a2fd4aa195d2033e47b8e64f" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 44935 } }
I20260929 10:32:50.276477 10494 leader_election.cc:302] T 00000000000000000000000000000000 P 2abed552a2fd4aa195d2033e47b8e64f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2abed552a2fd4aa195d2033e47b8e64f; no voters:
I20260929 10:32:50.276847 10494 leader_election.cc:288] T 00000000000000000000000000000000 P 2abed552a2fd4aa195d2033e47b8e64f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:32:50.276919 10499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2abed552a2fd4aa195d2033e47b8e64f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:32:50.276917 20848 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:32:50.277122 10499 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2abed552a2fd4aa195d2033e47b8e64f [term 1 LEADER]: Becoming Leader. State: Replica: 2abed552a2fd4aa195d2033e47b8e64f, State: Running, Role: LEADER
I20260929 10:32:50.277431 10499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2abed552a2fd4aa195d2033e47b8e64f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abed552a2fd4aa195d2033e47b8e64f" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 44935 } }
I20260929 10:32:50.278205 10494 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2abed552a2fd4aa195d2033e47b8e64f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:32:50.279234 10501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2abed552a2fd4aa195d2033e47b8e64f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2abed552a2fd4aa195d2033e47b8e64f. Latest consensus state: current_term: 1 leader_uuid: "2abed552a2fd4aa195d2033e47b8e64f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abed552a2fd4aa195d2033e47b8e64f" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 44935 } } }
I20260929 10:32:50.279356 10501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2abed552a2fd4aa195d2033e47b8e64f [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:50.279847 10512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:32:50.279808 10500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2abed552a2fd4aa195d2033e47b8e64f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2abed552a2fd4aa195d2033e47b8e64f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abed552a2fd4aa195d2033e47b8e64f" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 44935 } } }
I20260929 10:32:50.280395 10500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2abed552a2fd4aa195d2033e47b8e64f [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:50.283411 10512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:32:50.285786 10497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:50.280236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42047 (local address 127.20.92.62:44935)
0929 10:32:50.280340 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:50.280345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:50.280809 (+ 464us) server_negotiation.cc:407] Connection header received
0929 10:32:50.280930 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:50.280933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:50.280992 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:50.281091 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:50.281604 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:50.282432 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:50.283665 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:50.283975 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:50.284073 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:50.285068 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:50.285072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:50.285075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:50.285088 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:50.285116 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:50.285130 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:50.285198 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:50.285535 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 10:32:50.285600 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 10:32:50.289263 10512 catalog_manager.cc:1347] Generated new cluster ID: 8e0b1375374f40348bc4e032dc74254d
I20260929 10:32:50.289330 10512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:32:50.309923 10512 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:32:50.310989 10512 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:32:50.317759 10512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2abed552a2fd4aa195d2033e47b8e64f: Generated new TSK 0
I20260929 10:32:50.318689 10512 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:32:50.330946 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 10426
2026-09-29T10:32:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1489 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:32:50 dist-test-slave-9cj5 krb5kdc[10525](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:32:50 dist-test-slave-9cj5 krb5kdc[10525](info): set up 2 sockets
Sep 29 10:32:50 dist-test-slave-9cj5 krb5kdc[10525](info): commencing operation
krb5kdc: starting...
W20260929 10:32:51.379362 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:32:51 dist-test-slave-9cj5 krb5kdc[10525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:32:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:32:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
I20260929 10:32:51.505467 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:40591
--webserver_interface=127.20.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:42433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:40591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:32:51.614809 10538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:51.615082 10538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:51.615125 10538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:51.619069 10538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:32:51.619138 10538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:51.619158 10538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:51.619174 10538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:32:51.619190 10538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:32:51.624578 10538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:42433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:40591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:40591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:51.625579 10538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:51.626561 10538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:51.635191 10546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:51.635177 10543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:51.635224 10544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:51.635380 10538 server_base.cc:1034] running on GCE node
I20260929 10:32:51.636089 10538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:51.637162 10538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:51.638352 10538 hybrid_clock.cc:648] HybridClock initialized: now 1790677971638330 us; error 32 us; skew 500 ppm
Sep 29 10:32:51 dist-test-slave-9cj5 krb5kdc[10525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677971, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:51.641330 10538 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:32:51.642367 10538 webserver.cc:460] Webserver started at http://127.20.92.62:46387/ using document root <none> and password file <none>
I20260929 10:32:51.642977 10538 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:51.643050 10538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:51.643285 10538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:51.645493 10538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "e11f5498d9924214b06f4258808e8d52"
format_stamp: "Formatted at 2026-09-29 10:32:51 on dist-test-slave-9cj5"
server_key: "a1ea2f1e39ec4ee49ed4f218c0b957ee"
server_key_iv: "ad0d71616fb4f6339835a606df02b3c8"
server_key_version: "encryptionkey@0"
I20260929 10:32:51.646176 10538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "e11f5498d9924214b06f4258808e8d52"
format_stamp: "Formatted at 2026-09-29 10:32:51 on dist-test-slave-9cj5"
server_key: "a1ea2f1e39ec4ee49ed4f218c0b957ee"
server_key_iv: "ad0d71616fb4f6339835a606df02b3c8"
server_key_version: "encryptionkey@0"
I20260929 10:32:51.650405 10538 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 10:32:51.653168 10553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:51.654587 10538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:32:51.654737 10538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "e11f5498d9924214b06f4258808e8d52"
format_stamp: "Formatted at 2026-09-29 10:32:51 on dist-test-slave-9cj5"
server_key: "a1ea2f1e39ec4ee49ed4f218c0b957ee"
server_key_iv: "ad0d71616fb4f6339835a606df02b3c8"
server_key_version: "encryptionkey@0"
I20260929 10:32:51.654848 10538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:51.659924 10538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:51.663064 10538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:51.663277 10538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:51.670820 10538 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:40591
I20260929 10:32:51.670888 10605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:40591 every 8 connection(s)
I20260929 10:32:51.671868 10538 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:32:51.675117 10606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:51.680723 10606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e11f5498d9924214b06f4258808e8d52: Bootstrap starting.
I20260929 10:32:51.681795 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 10538
I20260929 10:32:51.681944 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:32:51.682284 20848 external_mini_cluster.cc:1428] Setting key 8bc0053413c664ceb4fed832ea937dc4
I20260929 10:32:51.683676 10606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e11f5498d9924214b06f4258808e8d52: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:51.684373 10606 log.cc:824] T 00000000000000000000000000000000 P e11f5498d9924214b06f4258808e8d52: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:51.686371 10606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e11f5498d9924214b06f4258808e8d52: No bootstrap required, opened a new log
I20260929 10:32:51.687332 10609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:51.683408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58797 (local address 127.20.92.62:40591)
0929 10:32:51.683893 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:51.683907 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:51.683940 (+ 33us) server_negotiation.cc:407] Connection header received
0929 10:32:51.684646 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:51.684664 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:51.684928 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:51.685298 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:51.686120 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:51.686124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:51.686136 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:51.686146 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:51.686196 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:51.686214 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:51.686331 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:51.686426 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 10:32:51.686589 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":146,"threads_started":1}
I20260929 10:32:51.690167 20848 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:32:51.690654 10606 raft_consensus.cc:348] T 00000000000000000000000000000000 P e11f5498d9924214b06f4258808e8d52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11f5498d9924214b06f4258808e8d52" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 40591 } }
I20260929 10:32:51.690928 10606 raft_consensus.cc:374] T 00000000000000000000000000000000 P e11f5498d9924214b06f4258808e8d52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:32:51.691004 10606 raft_consensus.cc:729] T 00000000000000000000000000000000 P e11f5498d9924214b06f4258808e8d52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e11f5498d9924214b06f4258808e8d52, State: Initialized, Role: FOLLOWER
I20260929 10:32:51.691465 10606 consensus_queue.cc:260] T 00000000000000000000000000000000 P e11f5498d9924214b06f4258808e8d52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11f5498d9924214b06f4258808e8d52" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 40591 } }
I20260929 10:32:51.691659 10606 raft_consensus.cc:388] T 00000000000000000000000000000000 P e11f5498d9924214b06f4258808e8d52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:32:51.691735 10606 raft_consensus.cc:482] T 00000000000000000000000000000000 P e11f5498d9924214b06f4258808e8d52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:32:51.691834 10606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e11f5498d9924214b06f4258808e8d52 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:51.693111 10606 raft_consensus.cc:504] T 00000000000000000000000000000000 P e11f5498d9924214b06f4258808e8d52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11f5498d9924214b06f4258808e8d52" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 40591 } }
I20260929 10:32:51.693493 10606 leader_election.cc:302] T 00000000000000000000000000000000 P e11f5498d9924214b06f4258808e8d52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e11f5498d9924214b06f4258808e8d52; no voters:
I20260929 10:32:51.693861 10606 leader_election.cc:288] T 00000000000000000000000000000000 P e11f5498d9924214b06f4258808e8d52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:32:51.693919 10616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e11f5498d9924214b06f4258808e8d52 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:32:51.694110 10616 raft_consensus.cc:686] T 00000000000000000000000000000000 P e11f5498d9924214b06f4258808e8d52 [term 1 LEADER]: Becoming Leader. State: Replica: e11f5498d9924214b06f4258808e8d52, State: Running, Role: LEADER
I20260929 10:32:51.694289 10609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:51.692402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44095 (local address 127.20.92.62:40591)
0929 10:32:51.692539 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:51.692544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:51.692779 (+ 235us) server_negotiation.cc:407] Connection header received
0929 10:32:51.692882 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:51.692884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:51.692913 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:51.692983 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:51.693740 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:51.693786 (+ 46us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:51.693789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:32:51.693803 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:51.693832 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:32:51.693844 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:51.693950 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:51.694121 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 10:32:51.694160 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 10:32:51.694401 10616 consensus_queue.cc:237] T 00000000000000000000000000000000 P e11f5498d9924214b06f4258808e8d52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11f5498d9924214b06f4258808e8d52" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 40591 } }
I20260929 10:32:51.695117 10606 sys_catalog.cc:564] T 00000000000000000000000000000000 P e11f5498d9924214b06f4258808e8d52 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:32:51.695549 10616 sys_catalog.cc:455] T 00000000000000000000000000000000 P e11f5498d9924214b06f4258808e8d52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e11f5498d9924214b06f4258808e8d52. Latest consensus state: current_term: 1 leader_uuid: "e11f5498d9924214b06f4258808e8d52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11f5498d9924214b06f4258808e8d52" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 40591 } } }
I20260929 10:32:51.695664 10616 sys_catalog.cc:458] T 00000000000000000000000000000000 P e11f5498d9924214b06f4258808e8d52 [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:51.695866 10616 sys_catalog.cc:455] T 00000000000000000000000000000000 P e11f5498d9924214b06f4258808e8d52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e11f5498d9924214b06f4258808e8d52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11f5498d9924214b06f4258808e8d52" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 40591 } } }
I20260929 10:32:51.695966 10616 sys_catalog.cc:458] T 00000000000000000000000000000000 P e11f5498d9924214b06f4258808e8d52 [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:51.696094 10620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:32:51.698828 10620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:32:51.704346 10620 catalog_manager.cc:1347] Generated new cluster ID: 6f557210adbe465195350c660b2c35eb
I20260929 10:32:51.704414 10620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:32:51.736750 10620 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:32:51.737121 10620 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:32:51.744554 10620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e11f5498d9924214b06f4258808e8d52: Generated new TSK 0
I20260929 10:32:51.745350 10620 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:32:51.759543 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 10538
2026-09-29T10:32:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1427 ms)
[----------] 18 tests from AuthTokenIssuingTest (25906 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:32:51 dist-test-slave-9cj5 krb5kdc[10637](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:32:51 dist-test-slave-9cj5 krb5kdc[10637](info): set up 2 sockets
Sep 29 10:32:51 dist-test-slave-9cj5 krb5kdc[10637](info): commencing operation
krb5kdc: starting...
W20260929 10:32:52.815040 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:32:52 dist-test-slave-9cj5 krb5kdc[10637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:32:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:32:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
I20260929 10:32:52.936573 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:34557
--webserver_interface=127.20.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:33877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:34557,127.20.92.61:37187,127.20.92.60:40397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:32:53.057548 10650 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 10:32:53.057878 10650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:53.057905 10650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:53.058022 10650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:53.062196 10650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:32:53.062266 10650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:53.062287 10650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:53.062307 10650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:32:53.062323 10650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:32:53.067870 10650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:33877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:34557,127.20.92.61:37187,127.20.92.60:40397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:34557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:53.069036 10650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:53.070093 10650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:53.078655 10656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:53.078735 10655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:53.078732 10650 server_base.cc:1034] running on GCE node
W20260929 10:32:53.078904 10658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:53.079551 10650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:53.080717 10650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:53.081938 10650 hybrid_clock.cc:648] HybridClock initialized: now 1790677973081910 us; error 48 us; skew 500 ppm
Sep 29 10:32:53 dist-test-slave-9cj5 krb5kdc[10637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677973, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:53.090742 10650 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:32:53.091691 10650 webserver.cc:460] Webserver started at http://127.20.92.62:39823/ using document root <none> and password file <none>
I20260929 10:32:53.092291 10650 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:53.092386 10650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:53.092631 10650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:53.094801 10650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "d03cd4157eee44e9a6699b03cf41b321"
format_stamp: "Formatted at 2026-09-29 10:32:53 on dist-test-slave-9cj5"
server_key: "08c70906694156b1faf8b83be0b7a10f"
server_key_iv: "f93ffd889f83a864c5e142fa12b74b84"
server_key_version: "encryptionkey@0"
I20260929 10:32:53.095376 10650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "d03cd4157eee44e9a6699b03cf41b321"
format_stamp: "Formatted at 2026-09-29 10:32:53 on dist-test-slave-9cj5"
server_key: "08c70906694156b1faf8b83be0b7a10f"
server_key_iv: "f93ffd889f83a864c5e142fa12b74b84"
server_key_version: "encryptionkey@0"
I20260929 10:32:53.099231 10650 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 10:32:53.101737 10665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:53.103124 10650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:32:53.103271 10650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "d03cd4157eee44e9a6699b03cf41b321"
format_stamp: "Formatted at 2026-09-29 10:32:53 on dist-test-slave-9cj5"
server_key: "08c70906694156b1faf8b83be0b7a10f"
server_key_iv: "f93ffd889f83a864c5e142fa12b74b84"
server_key_version: "encryptionkey@0"
I20260929 10:32:53.103390 10650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:53.118376 10650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:53.121539 10650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:53.121832 10650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:53.129494 10650 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:34557
I20260929 10:32:53.129822 10717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:34557 every 8 connection(s)
I20260929 10:32:53.130620 10650 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:32:53.133360 10718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:53.134016 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 10650
I20260929 10:32:53.134161 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:32:53.134438 20848 external_mini_cluster.cc:1428] Setting key 22ed232c436b7c9bd0d29211ca9d8b25
I20260929 10:32:53.137161 10718 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34557 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.92.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.61@KRBTEST.COM" created.
Sep 29 10:32:53 dist-test-slave-9cj5 krb5kdc[10637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677973, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:32:53.154093 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.139534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43864 (local address 127.20.92.62:34557)
0929 10:32:53.139972 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:53.139992 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:53.140272 (+ 280us) server_negotiation.cc:407] Connection header received
0929 10:32:53.141293 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:53.141297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:53.141473 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:53.141653 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:53.142495 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.143442 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.144402 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.144938 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.145107 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:53.148166 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:53.148176 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:53.148193 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:53.148225 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:53.150462 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:53.151025 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:53.151034 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:53.151037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:53.151220 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:53.151552 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:53.151557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:53.151561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:53.152135 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:53.152277 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:53.152714 (+ 437us) server_negotiation.cc:299] Negotiation successful
0929 10:32:53.152940 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":139,"threads_started":1}
I20260929 10:32:53.154100 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.139535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:34557 (local address 127.0.0.1:43864)
0929 10:32:53.140124 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:53.140160 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:53.140995 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:53.141651 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:53.141656 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:53.142116 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:53.142340 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:53.142352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.143566 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.143569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:53.144279 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:53.144289 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.145136 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.145144 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:53.145287 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:53.145971 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:32:53.145998 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:53.148001 (+ 2003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:53.150606 (+ 2605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:53.150614 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:53.150623 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:53.150877 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:53.151330 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:53.151333 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:53.151334 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:53.151444 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:53.152300 (+ 856us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:53.152306 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:53.152435 (+ 129us) client_negotiation.cc:769] Sending connection context
0929 10:32:53.152596 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 10:32:53.153070 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":116,"spinlock_wait_cycles":5504,"thread_start_us":168,"threads_started":1}
I20260929 10:32:53.156904 10718 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.92.61" port: 37187 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:32:53.157691 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.157381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:37187 (local address 127.0.0.1:58978)
0929 10:32:53.157528 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:53.157601 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.61:37187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 10:32:53.157998 10669 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.92.61:37187: connect: Connection refused (error 111)
I20260929 10:32:53.158890 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.158639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:37187 (local address 127.0.0.1:58990)
0929 10:32:53.158792 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:53.158837 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.61:37187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260929 10:32:53.159088 10718 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.92.61:37187: Network error: Client connection negotiation failed: client connection to 127.20.92.61:37187: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.61.keytab.
Entry for principal kudu/127.20.92.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.61.keytab.
I20260929 10:32:53.163398 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.61:37187
--webserver_interface=127.20.92.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.61.keytab
--principal=kudu/127.20.92.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:33877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:34557,127.20.92.61:37187,127.20.92.60:40397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:32:53.208333 10718 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.92.61" port: 37187 } attempt: 1
I20260929 10:32:53.209204 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.208870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:37187 (local address 127.0.0.1:58994)
0929 10:32:53.209036 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:53.209114 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.61:37187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 10:32:53.210088 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.209845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:37187 (local address 127.0.0.1:59000)
0929 10:32:53.209963 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:53.210014 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.61:37187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 10:32:53.210256 10718 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.92.61:37187: Network error: Client connection negotiation failed: client connection to 127.20.92.61:37187: connect: Connection refused (error 111)
W20260929 10:32:53.275442 10725 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 10:32:53.275738 10725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:53.275797 10725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:53.275879 10725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 10:32:53.278512 10718 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.92.61" port: 37187 } attempt: 2
I20260929 10:32:53.279388 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.279055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:37187 (local address 127.0.0.1:59006)
0929 10:32:53.279229 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:53.279299 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.61:37187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 10:32:53.280134 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.279910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:37187 (local address 127.0.0.1:59008)
0929 10:32:53.280015 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:53.280065 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.61:37187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 10:32:53.280280 10718 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.92.61:37187: Network error: Client connection negotiation failed: client connection to 127.20.92.61:37187: connect: Connection refused (error 111)
W20260929 10:32:53.280614 10725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:32:53.280689 10725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:53.280733 10725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:53.280769 10725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:32:53.280804 10725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:32:53.286546 10725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:33877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:34557,127.20.92.61:37187,127.20.92.60:40397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.61.keytab
--principal=kudu/127.20.92.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.61:37187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.92.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:53.287760 10725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:53.288844 10725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:53.297564 10731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:53.297575 10730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:53.297536 10733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:53.297883 10725 server_base.cc:1034] running on GCE node
I20260929 10:32:53.298504 10725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:53.299618 10725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:53.300812 10725 hybrid_clock.cc:648] HybridClock initialized: now 1790677973300798 us; error 26 us; skew 500 ppm
Sep 29 10:32:53 dist-test-slave-9cj5 krb5kdc[10637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677973, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:53.303627 10725 init.cc:377] Logged in from keytab as kudu/127.20.92.61@KRBTEST.COM (short username kudu)
I20260929 10:32:53.304621 10725 webserver.cc:460] Webserver started at http://127.20.92.61:45151/ using document root <none> and password file <none>
I20260929 10:32:53.305210 10725 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:53.305306 10725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:53.305567 10725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:53.308043 10725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-1/data/instance:
uuid: "4fb929af0af84e16b7f497a28dd12dc9"
format_stamp: "Formatted at 2026-09-29 10:32:53 on dist-test-slave-9cj5"
server_key: "a87b4ed6a81f5abfe862b666cc279587"
server_key_iv: "5b96ba968ba67c4a047ed89f39ae4436"
server_key_version: "encryptionkey@0"
I20260929 10:32:53.308652 10725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-1/wal/instance:
uuid: "4fb929af0af84e16b7f497a28dd12dc9"
format_stamp: "Formatted at 2026-09-29 10:32:53 on dist-test-slave-9cj5"
server_key: "a87b4ed6a81f5abfe862b666cc279587"
server_key_iv: "5b96ba968ba67c4a047ed89f39ae4436"
server_key_version: "encryptionkey@0"
I20260929 10:32:53.312906 10725 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260929 10:32:53.315490 10740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:53.316788 10725 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:32:53.316936 10725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-1/wal
uuid: "4fb929af0af84e16b7f497a28dd12dc9"
format_stamp: "Formatted at 2026-09-29 10:32:53 on dist-test-slave-9cj5"
server_key: "a87b4ed6a81f5abfe862b666cc279587"
server_key_iv: "5b96ba968ba67c4a047ed89f39ae4436"
server_key_version: "encryptionkey@0"
I20260929 10:32:53.317059 10725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:53.352542 10725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:53.355849 10725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:53.356073 10725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:53.363642 10725 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.61:37187
I20260929 10:32:53.364058 10792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.61:37187 every 8 connection(s)
I20260929 10:32:53.364687 10725 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-1/data/info.pb
I20260929 10:32:53.367565 10793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:53.370077 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 10725
I20260929 10:32:53.370260 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-1/wal/instance
I20260929 10:32:53.370579 20848 external_mini_cluster.cc:1428] Setting key 825164fc82357095c2489c4ce60dbfad
I20260929 10:32:53.371272 10793 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34557 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:32:53.371471 10718 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.92.61" port: 37187 } attempt: 3
WARNING: no policy specified for kudu/127.20.92.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.60@KRBTEST.COM" created.
Sep 29 10:32:53 dist-test-slave-9cj5 krb5kdc[10637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677973, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.61@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
Sep 29 10:32:53 dist-test-slave-9cj5 krb5kdc[10637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677973, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for kudu/127.20.92.61@KRBTEST.COM
I20260929 10:32:53.389456 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.372037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:37187 (local address 127.0.0.1:59018)
0929 10:32:53.372154 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:53.372171 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:53.372275 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:53.374995 (+ 2720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:53.375002 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:53.375247 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:53.375425 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:53.375437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.377658 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.377662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:53.378484 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:53.378494 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.379303 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.379311 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:53.379456 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:53.380020 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:32:53.380059 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:53.384899 (+ 4840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:53.387582 (+ 2683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:53.387585 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:53.387588 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:53.387791 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:53.388194 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:53.388198 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:53.388200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:53.388286 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:53.388898 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:53.388905 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:53.389030 (+ 125us) client_negotiation.cc:769] Sending connection context
0929 10:32:53.389114 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 10:32:53.389259 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":2}
I20260929 10:32:53.390324 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.373568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43876 (local address 127.20.92.62:34557)
0929 10:32:53.373704 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:53.373709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:53.374091 (+ 382us) server_negotiation.cc:407] Connection header received
0929 10:32:53.374739 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:53.374743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:53.374801 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:53.374911 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:53.376114 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.376698 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.378152 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.378690 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.378857 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:53.384472 (+ 5615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:53.384476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:53.384478 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:53.384527 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:53.386597 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:53.387344 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:53.387349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:53.387351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:53.387446 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:53.387916 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:53.387921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:53.387924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:53.388194 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:53.388287 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:53.389894 (+ 1607us) server_negotiation.cc:299] Negotiation successful
0929 10:32:53.390061 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":53}
I20260929 10:32:53.390522 10797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.373550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:34557 (local address 127.0.0.1:43876)
0929 10:32:53.373969 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:53.374014 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:53.374484 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:53.374984 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:53.374996 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:53.375458 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:53.375961 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:53.375973 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.376820 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.376827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:53.378006 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:53.378017 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.378868 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.378873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:53.379059 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:53.379834 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:32:53.379864 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:53.384284 (+ 4420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:53.386729 (+ 2445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:53.386742 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:53.386757 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:53.387198 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:53.387570 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:53.387577 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:53.387582 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:53.387788 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:53.388349 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:53.388360 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:53.388767 (+ 407us) client_negotiation.cc:769] Sending connection context
0929 10:32:53.388960 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 10:32:53.389327 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":135,"spinlock_wait_cycles":896,"thread_start_us":132,"threads_started":1}
I20260929 10:32:53.390522 10796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.372744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59018 (local address 127.20.92.61:37187)
0929 10:32:53.373168 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:53.373188 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:53.373229 (+ 41us) server_negotiation.cc:407] Connection header received
0929 10:32:53.374473 (+ 1244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:53.374501 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:53.374769 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:53.375627 (+ 858us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:53.375825 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.377506 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.378612 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.379150 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.379282 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:53.385127 (+ 5845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:53.385138 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:53.385157 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:53.385185 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:53.387449 (+ 2264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:53.387954 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:53.387975 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:53.387976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:53.388057 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:53.388400 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:53.388404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:53.388405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:53.388758 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:53.389863 (+ 1105us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:53.389989 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 10:32:53.390143 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":285,"spinlock_wait_cycles":3456,"thread_start_us":150,"threads_started":1}
I20260929 10:32:53.392462 10793 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.92.61" port: 37187 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:32:53.392886 10718 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40397 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:32:53.393666 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.393348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:40397 (local address 127.0.0.1:58364)
0929 10:32:53.393555 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:53.393602 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260929 10:32:53.394476 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.394177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:40397 (local address 127.0.0.1:58368)
0929 10:32:53.394363 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:53.394425 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260929 10:32:53.394618 10718 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.92.60:40397: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40397: connect: Connection refused (error 111)
Sep 29 10:32:53 dist-test-slave-9cj5 krb5kdc[10637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677973, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.61@KRBTEST.COM for kudu/127.20.92.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.60.keytab.
Entry for principal kudu/127.20.92.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.60.keytab.
I20260929 10:32:53.401332 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.60:40397
--webserver_interface=127.20.92.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.60.keytab
--principal=kudu/127.20.92.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:33877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:34557,127.20.92.61:37187,127.20.92.60:40397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:32:53.402060 10797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.393059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:37187 (local address 127.0.0.1:59026)
0929 10:32:53.393280 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:53.393295 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:53.393377 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:53.393957 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:53.393960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:53.394141 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:53.394269 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:53.394277 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.395393 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.395396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:53.396159 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:53.396169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.396776 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.396778 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:53.396853 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:53.397373 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:32:53.397384 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:53.398821 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:53.400498 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:53.400502 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:53.400504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:53.400748 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:53.401039 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:53.401044 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:53.401046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:53.401145 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:53.401510 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:53.401515 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:53.401626 (+ 111us) client_negotiation.cc:769] Sending connection context
0929 10:32:53.401736 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 10:32:53.401896 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260929 10:32:53.402087 10796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.393142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59026 (local address 127.20.92.61:37187)
0929 10:32:53.393683 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:53.393687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:53.393699 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:32:53.393728 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:53.393731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:53.393834 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:53.393948 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:53.394384 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.395264 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.396308 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.396661 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.396754 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:53.398943 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:53.398946 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:53.398948 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:53.398969 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:53.400393 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:53.400850 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:53.400852 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:53.400853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:53.400924 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:53.401237 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:53.401239 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:53.401240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:53.401393 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:53.401504 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:53.401801 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 10:32:53.401951 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":472}
I20260929 10:32:53.403647 10793 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40397 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:32:53.404472 10797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.404135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:40397 (local address 127.0.0.1:58376)
0929 10:32:53.404281 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:53.404385 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 10:32:53.404767 10744 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40397: connect: Connection refused (error 111)
I20260929 10:32:53.405622 10797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.405424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:40397 (local address 127.0.0.1:58390)
0929 10:32:53.405531 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:53.405574 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 10:32:53.405910 10793 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.92.60:40397: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40397: connect: Connection refused (error 111)
I20260929 10:32:53.425834 10718 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40397 } attempt: 1
I20260929 10:32:53.426738 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.426399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:40397 (local address 127.0.0.1:58394)
0929 10:32:53.426600 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:53.426663 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 10:32:53.427495 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.427267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:40397 (local address 127.0.0.1:58396)
0929 10:32:53.427393 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:53.427441 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 10:32:53.427747 10718 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.92.60:40397: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40397: connect: Connection refused (error 111)
I20260929 10:32:53.455211 10793 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40397 } attempt: 1
I20260929 10:32:53.456171 10797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.455875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:40397 (local address 127.0.0.1:58402)
0929 10:32:53.456036 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:53.456097 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 10:32:53.457006 10797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.456775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:40397 (local address 127.0.0.1:58418)
0929 10:32:53.456898 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:53.456950 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 10:32:53.457230 10793 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.92.60:40397: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40397: connect: Connection refused (error 111)
I20260929 10:32:53.502938 10718 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40397 } attempt: 2
I20260929 10:32:53.503815 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.503533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:40397 (local address 127.0.0.1:58432)
0929 10:32:53.503681 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:53.503742 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 10:32:53.504688 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.504393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:40397 (local address 127.0.0.1:58438)
0929 10:32:53.504573 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:53.504627 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 10:32:53.504871 10718 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.92.60:40397: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40397: connect: Connection refused (error 111)
W20260929 10:32:53.512770 10799 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 10:32:53.513214 10799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:53.513268 10799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:53.513336 10799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:53.517496 10799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:32:53.517578 10799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:53.517616 10799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:53.517652 10799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:32:53.517684 10799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:32:53.523130 10799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:33877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:34557,127.20.92.61:37187,127.20.92.60:40397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.60.keytab
--principal=kudu/127.20.92.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.60:40397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.92.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:53.524199 10799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:53.525203 10799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:32:53.525416 10793 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40397 } attempt: 2
I20260929 10:32:53.526279 10797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.525979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:40397 (local address 127.0.0.1:58444)
0929 10:32:53.526146 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:53.526206 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 10:32:53.527014 10797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.526792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:40397 (local address 127.0.0.1:58454)
0929 10:32:53.526910 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:53.526962 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 10:32:53.527173 10793 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.92.60:40397: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40397: connect: Connection refused (error 111)
W20260929 10:32:53.534169 10805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:53.534162 10808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:53.534162 10806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:53.534323 10799 server_base.cc:1034] running on GCE node
I20260929 10:32:53.534951 10799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:53.536170 10799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:53.537380 10799 hybrid_clock.cc:648] HybridClock initialized: now 1790677973537367 us; error 27 us; skew 500 ppm
Sep 29 10:32:53 dist-test-slave-9cj5 krb5kdc[10637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677973, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:53.540657 10799 init.cc:377] Logged in from keytab as kudu/127.20.92.60@KRBTEST.COM (short username kudu)
I20260929 10:32:53.541728 10799 webserver.cc:460] Webserver started at http://127.20.92.60:43947/ using document root <none> and password file <none>
I20260929 10:32:53.542367 10799 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:53.542429 10799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:53.542670 10799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:53.544855 10799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-2/data/instance:
uuid: "033c07cc0bfd44f3b7d1969c892450cc"
format_stamp: "Formatted at 2026-09-29 10:32:53 on dist-test-slave-9cj5"
server_key: "760eca62f2fab7aab78ce1c2ae9b8cd7"
server_key_iv: "950332b7c28fd62e1f9baa65d3f48f75"
server_key_version: "encryptionkey@0"
I20260929 10:32:53.545459 10799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-2/wal/instance:
uuid: "033c07cc0bfd44f3b7d1969c892450cc"
format_stamp: "Formatted at 2026-09-29 10:32:53 on dist-test-slave-9cj5"
server_key: "760eca62f2fab7aab78ce1c2ae9b8cd7"
server_key_iv: "950332b7c28fd62e1f9baa65d3f48f75"
server_key_version: "encryptionkey@0"
I20260929 10:32:53.549408 10799 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260929 10:32:53.551914 10815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:53.553213 10799 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:32:53.553341 10799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-2/wal
uuid: "033c07cc0bfd44f3b7d1969c892450cc"
format_stamp: "Formatted at 2026-09-29 10:32:53 on dist-test-slave-9cj5"
server_key: "760eca62f2fab7aab78ce1c2ae9b8cd7"
server_key_iv: "950332b7c28fd62e1f9baa65d3f48f75"
server_key_version: "encryptionkey@0"
I20260929 10:32:53.553452 10799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:53.573896 10799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:53.577409 10799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:53.577629 10799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:53.585407 10799 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.60:40397
I20260929 10:32:53.585731 10867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.60:40397 every 8 connection(s)
I20260929 10:32:53.586501 10799 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-2/data/info.pb
I20260929 10:32:53.588992 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 10799
I20260929 10:32:53.589131 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677679287822-20848-0/minicluster-data/master-2/wal/instance
I20260929 10:32:53.589378 10868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:53.589455 20848 external_mini_cluster.cc:1428] Setting key 5c24e048d8d09d809da6cbe884b1a6fd
I20260929 10:32:53.594293 10868 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34557 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 10:32:53 dist-test-slave-9cj5 krb5kdc[10637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.60@KRBTEST.COM
Sep 29 10:32:53 dist-test-slave-9cj5 krb5kdc[10637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677973, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.60@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:32:53.604068 10718 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40397 } attempt: 3
I20260929 10:32:53.604703 10871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.590825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58456 (local address 127.20.92.60:40397)
0929 10:32:53.591245 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:53.591262 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:53.591297 (+ 35us) server_negotiation.cc:407] Connection header received
0929 10:32:53.592053 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:53.592070 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:53.592391 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:53.592682 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:53.593624 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.594892 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.595982 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.596687 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.596855 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:53.599466 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:53.599471 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:53.599479 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:53.599503 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:53.601365 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:53.601960 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:53.601966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:53.601972 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:53.602097 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:53.602428 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:53.602432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:53.602433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:53.602989 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:53.603232 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:53.603492 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 10:32:53.603770 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":148,"threads_started":1}
I20260929 10:32:53.605458 10872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.596050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:34557 (local address 127.0.0.1:43886)
0929 10:32:53.596400 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:53.596445 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:53.596560 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:53.596881 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:53.596887 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:53.597365 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:53.597529 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:53.597536 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.598410 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.598412 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:53.599119 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:53.599129 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.599681 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.599685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:53.599811 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:53.600250 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:32:53.600269 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:53.601891 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:53.603747 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:53.603763 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:53.603768 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:53.603994 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:53.604325 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:53.604328 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:53.604330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:53.604416 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:53.604899 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:53.604905 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:53.605035 (+ 130us) client_negotiation.cc:769] Sending connection context
0929 10:32:53.605175 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 10:32:53.605331 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":130,"threads_started":1}
I20260929 10:32:53.605563 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.596094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43886 (local address 127.20.92.62:34557)
0929 10:32:53.596245 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:53.596249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:53.596563 (+ 314us) server_negotiation.cc:407] Connection header received
0929 10:32:53.596668 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:53.596671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:53.596721 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:53.596835 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:53.597661 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.598254 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.599246 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.599568 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.599652 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:53.602018 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:53.602022 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:53.602024 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:53.602049 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:53.603633 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:53.604124 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:53.604127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:53.604129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:53.604214 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:53.604519 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:53.604522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:53.604523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:53.604742 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:53.604818 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:53.605276 (+ 458us) server_negotiation.cc:299] Negotiation successful
0929 10:32:53.605398 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 10:32:53.607323 10868 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.92.61" port: 37187 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 10:32:53 dist-test-slave-9cj5 krb5kdc[10637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677973, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for kudu/127.20.92.60@KRBTEST.COM
Sep 29 10:32:53 dist-test-slave-9cj5 krb5kdc[10637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
Sep 29 10:32:53 dist-test-slave-9cj5 krb5kdc[10637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677973, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.60@KRBTEST.COM for kudu/127.20.92.61@KRBTEST.COM
I20260929 10:32:53.615928 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.604573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:40397 (local address 127.0.0.1:58468)
0929 10:32:53.604770 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:53.604787 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:53.604870 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:53.606167 (+ 1297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:53.606170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:53.606345 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:53.606444 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:53.606452 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.607524 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.607527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:53.608384 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:53.608395 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.609007 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.609011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:53.609108 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:53.610412 (+ 1304us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:32:53.610423 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:53.611817 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:53.614030 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:53.614034 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:53.614036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:53.614301 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:53.614711 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:53.614714 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:53.614718 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:53.614847 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:53.615361 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:53.615366 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:53.615476 (+ 110us) client_negotiation.cc:769] Sending connection context
0929 10:32:53.615568 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 10:32:53.615714 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260929 10:32:53.616532 10873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.604612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58468 (local address 127.20.92.60:40397)
0929 10:32:53.605876 (+ 1264us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:53.605888 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:53.605901 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:32:53.605940 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:53.605946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:53.605995 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:53.606151 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:53.606578 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.607386 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.608507 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.608861 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.609035 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:53.611928 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:53.611931 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:53.611934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:53.611965 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:53.613915 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:53.614413 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:53.614419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:53.614425 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:53.614536 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:53.614977 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:53.614981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:53.614985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:53.615227 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:53.616115 (+ 888us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:53.616199 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 10:32:53.616314 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":1181,"thread_start_us":118,"threads_started":1}
I20260929 10:32:53.617712 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.607123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43896 (local address 127.20.92.62:34557)
0929 10:32:53.607660 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:53.607664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:53.607676 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:32:53.607706 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:53.607709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:53.607798 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:53.607872 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:53.608618 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.609342 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.610397 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.610750 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.610861 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:53.613630 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:53.613635 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:53.613638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:53.613667 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:53.615576 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:53.616016 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:53.616020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:53.616022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:53.616109 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:53.616418 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:53.616421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:53.616423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:53.616671 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:53.616905 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:53.617403 (+ 498us) server_negotiation.cc:299] Negotiation successful
0929 10:32:53.617514 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464}
I20260929 10:32:53.618327 10793 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40397 } attempt: 3
I20260929 10:32:53.619412 10872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.607798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:37187 (local address 127.0.0.1:59036)
0929 10:32:53.607948 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:53.607961 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:53.608040 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:53.608817 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:53.608820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:53.608984 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:53.609128 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:53.609134 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.610170 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.610173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:53.611705 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:53.611720 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.612570 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.612575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:53.612675 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:53.613282 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:32:53.613300 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:53.614819 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:53.617214 (+ 2395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:53.617217 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:53.617219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:53.617452 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:53.617992 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:53.617995 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:53.617996 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:53.618123 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:53.618711 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:53.618716 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:53.619019 (+ 303us) client_negotiation.cc:769] Sending connection context
0929 10:32:53.619140 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 10:32:53.619260 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":34}
I20260929 10:32:53.619433 10796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.607886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59036 (local address 127.20.92.61:37187)
0929 10:32:53.608607 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:53.608613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:53.608626 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:32:53.608664 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:53.608667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:53.608713 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:53.608803 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:53.609278 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.610059 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.611848 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.612307 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.612426 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:53.614947 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:53.614951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:53.614953 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:53.614988 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:53.616583 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:53.617569 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:53.617572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:53.617574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:53.617659 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:53.618251 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:53.618257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:53.618261 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:53.618490 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:53.618589 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:53.619196 (+ 607us) server_negotiation.cc:299] Negotiation successful
0929 10:32:53.619302 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617}
I20260929 10:32:53.620239 10718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d03cd4157eee44e9a6699b03cf41b321: Bootstrap starting.
I20260929 10:32:53.622380 10868 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40397 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:32:53.623770 10718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d03cd4157eee44e9a6699b03cf41b321: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:53.624818 10718 log.cc:824] T 00000000000000000000000000000000 P d03cd4157eee44e9a6699b03cf41b321: Log is configured to *not* fsync() on all Append() calls
Sep 29 10:32:53 dist-test-slave-9cj5 krb5kdc[10637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.61@KRBTEST.COM
Sep 29 10:32:53 dist-test-slave-9cj5 krb5kdc[10637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677973, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.61@KRBTEST.COM for kudu/127.20.92.60@KRBTEST.COM
I20260929 10:32:53.627316 10718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d03cd4157eee44e9a6699b03cf41b321: No bootstrap required, opened a new log
Sep 29 10:32:53 dist-test-slave-9cj5 krb5kdc[10637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677973, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.60@KRBTEST.COM for kudu/127.20.92.60@KRBTEST.COM
I20260929 10:32:53.631230 10797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.618697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:40397 (local address 127.0.0.1:58478)
0929 10:32:53.618792 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:53.618806 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:53.618875 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:53.622190 (+ 3315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:53.622193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:53.622361 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:53.622449 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:53.622458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.623701 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.623705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:53.624511 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:53.624521 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.625224 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.625227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:53.625308 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:53.625833 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:32:53.625848 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:53.627504 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:53.629561 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:53.629564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:53.629566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:53.629815 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:53.630124 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:53.630129 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:53.630134 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:53.630228 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:53.630689 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:53.630692 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:53.630795 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 10:32:53.630885 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 10:32:53.631028 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 10:32:53.631633 10796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.619181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59044 (local address 127.20.92.61:37187)
0929 10:32:53.619652 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:53.619656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:53.619668 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:32:53.619711 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:53.619714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:53.619748 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:53.619825 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:53.620669 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.621154 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.622342 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.622711 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.622794 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:53.626322 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:53.626327 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:53.626329 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:53.626356 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:53.629251 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:53.630099 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:53.630106 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:53.630108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:53.630198 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:53.630531 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:53.630535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:53.630537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:53.630748 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:53.630955 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:53.631325 (+ 370us) server_negotiation.cc:299] Negotiation successful
0929 10:32:53.631439 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":110,"threads_started":1}
I20260929 10:32:53.631780 10873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.619068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58478 (local address 127.20.92.60:40397)
0929 10:32:53.621998 (+ 2930us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:53.622002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:53.622013 (+ 11us) server_negotiation.cc:407] Connection header received
0929 10:32:53.622045 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:53.622048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:53.622087 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:53.622162 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:53.622631 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.623334 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.624698 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.625089 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.625558 (+ 469us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:53.627653 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:53.627657 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:53.627659 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:53.627683 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:53.629465 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:53.629911 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:53.629916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:53.629920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:53.630004 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:53.630351 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:53.630354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:53.630356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:53.630567 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:53.631418 (+ 851us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:53.631485 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 10:32:53.631621 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2860}
I20260929 10:32:53.632439 10718 raft_consensus.cc:348] T 00000000000000000000000000000000 P d03cd4157eee44e9a6699b03cf41b321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03cd4157eee44e9a6699b03cf41b321" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34557 } } peers { permanent_uuid: "4fb929af0af84e16b7f497a28dd12dc9" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 37187 } } peers { permanent_uuid: "033c07cc0bfd44f3b7d1969c892450cc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40397 } }
I20260929 10:32:53.632696 10718 raft_consensus.cc:729] T 00000000000000000000000000000000 P d03cd4157eee44e9a6699b03cf41b321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d03cd4157eee44e9a6699b03cf41b321, State: Initialized, Role: FOLLOWER
I20260929 10:32:53.632725 10872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.622810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:40397 (local address 127.0.0.1:58486)
0929 10:32:53.623482 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:53.623501 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:53.623586 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:53.623842 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:53.623844 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:53.624003 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:53.624072 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:53.624078 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.625004 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.625007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:53.625833 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:53.625843 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.626506 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.626510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:53.626586 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:53.627003 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:32:53.627021 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:53.628444 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:53.630669 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:53.630674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:53.630675 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:53.630895 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:53.631165 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:53.631168 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:53.631170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:53.631238 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:53.632277 (+ 1039us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:53.632280 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:53.632365 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 10:32:53.632460 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 10:32:53.632576 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610}
I20260929 10:32:53.633185 10871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.623023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58486 (local address 127.20.92.60:40397)
0929 10:32:53.623141 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:53.623145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:53.623554 (+ 409us) server_negotiation.cc:407] Connection header received
0929 10:32:53.623655 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:53.623660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:53.623700 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:53.623764 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:53.624180 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.624905 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.625956 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.626292 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.626403 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:53.628552 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:53.628555 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:53.628558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:53.628580 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:53.630562 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:53.630986 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:53.630992 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:53.630995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:53.631064 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:53.631559 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:53.631565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:53.631569 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:53.632175 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:53.632285 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:53.632963 (+ 678us) server_negotiation.cc:299] Negotiation successful
0929 10:32:53.633063 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 10:32:53.633765 10718 consensus_queue.cc:260] T 00000000000000000000000000000000 P d03cd4157eee44e9a6699b03cf41b321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03cd4157eee44e9a6699b03cf41b321" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34557 } } peers { permanent_uuid: "4fb929af0af84e16b7f497a28dd12dc9" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 37187 } } peers { permanent_uuid: "033c07cc0bfd44f3b7d1969c892450cc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40397 } }
I20260929 10:32:53.634930 10793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fb929af0af84e16b7f497a28dd12dc9: Bootstrap starting.
I20260929 10:32:53.635352 10718 sys_catalog.cc:564] T 00000000000000000000000000000000 P d03cd4157eee44e9a6699b03cf41b321 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:32:53.635821 10868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 033c07cc0bfd44f3b7d1969c892450cc: Bootstrap starting.
I20260929 10:32:53.635931 20848 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:32:53.637382 10692 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d03cd4157eee44e9a6699b03cf41b321"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43896
I20260929 10:32:53.637590 10793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fb929af0af84e16b7f497a28dd12dc9: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:53.638154 10868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 033c07cc0bfd44f3b7d1969c892450cc: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:53.637609 10692 raft_consensus.cc:482] T 00000000000000000000000000000000 P d03cd4157eee44e9a6699b03cf41b321 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 10:32:53.638290 10692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d03cd4157eee44e9a6699b03cf41b321 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:53.639020 10868 log.cc:824] T 00000000000000000000000000000000 P 033c07cc0bfd44f3b7d1969c892450cc: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:53.639575 10692 raft_consensus.cc:504] T 00000000000000000000000000000000 P d03cd4157eee44e9a6699b03cf41b321 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03cd4157eee44e9a6699b03cf41b321" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34557 } } peers { permanent_uuid: "4fb929af0af84e16b7f497a28dd12dc9" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 37187 } } peers { permanent_uuid: "033c07cc0bfd44f3b7d1969c892450cc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40397 } }
I20260929 10:32:53.640473 10692 leader_election.cc:288] T 00000000000000000000000000000000 P d03cd4157eee44e9a6699b03cf41b321 [CANDIDATE]: Term 1 election: Requested vote from peers 4fb929af0af84e16b7f497a28dd12dc9 (127.20.92.61:37187), 033c07cc0bfd44f3b7d1969c892450cc (127.20.92.60:40397)
I20260929 10:32:53.641162 10868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 033c07cc0bfd44f3b7d1969c892450cc: No bootstrap required, opened a new log
I20260929 10:32:53.641373 10876 sys_catalog.cc:455] T 00000000000000000000000000000000 P d03cd4157eee44e9a6699b03cf41b321 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03cd4157eee44e9a6699b03cf41b321" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34557 } } peers { permanent_uuid: "4fb929af0af84e16b7f497a28dd12dc9" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 37187 } } peers { permanent_uuid: "033c07cc0bfd44f3b7d1969c892450cc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40397 } } }
I20260929 10:32:53.641566 10876 sys_catalog.cc:458] T 00000000000000000000000000000000 P d03cd4157eee44e9a6699b03cf41b321 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:32:53.643800 10793 log.cc:824] T 00000000000000000000000000000000 P 4fb929af0af84e16b7f497a28dd12dc9: Log is configured to *not* fsync() on all Append() calls
W20260929 10:32:53.644480 10887 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d03cd4157eee44e9a6699b03cf41b321: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:32:53.644563 10887 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:32:53.645361 10868 raft_consensus.cc:348] T 00000000000000000000000000000000 P 033c07cc0bfd44f3b7d1969c892450cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03cd4157eee44e9a6699b03cf41b321" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34557 } } peers { permanent_uuid: "4fb929af0af84e16b7f497a28dd12dc9" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 37187 } } peers { permanent_uuid: "033c07cc0bfd44f3b7d1969c892450cc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40397 } }
I20260929 10:32:53.645614 10868 raft_consensus.cc:729] T 00000000000000000000000000000000 P 033c07cc0bfd44f3b7d1969c892450cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 033c07cc0bfd44f3b7d1969c892450cc, State: Initialized, Role: FOLLOWER
I20260929 10:32:53.645975 10868 consensus_queue.cc:260] T 00000000000000000000000000000000 P 033c07cc0bfd44f3b7d1969c892450cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03cd4157eee44e9a6699b03cf41b321" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34557 } } peers { permanent_uuid: "4fb929af0af84e16b7f497a28dd12dc9" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 37187 } } peers { permanent_uuid: "033c07cc0bfd44f3b7d1969c892450cc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40397 } }
I20260929 10:32:53.647264 10868 sys_catalog.cc:564] T 00000000000000000000000000000000 P 033c07cc0bfd44f3b7d1969c892450cc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:32:53.648121 10793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fb929af0af84e16b7f497a28dd12dc9: No bootstrap required, opened a new log
I20260929 10:32:53.643967 10650 master_runner.cc:186] Error getting master registration for 127.20.92.61:37187: OK, instance_id { permanent_uuid: "4fb929af0af84e16b7f497a28dd12dc9" instance_seqno: 1790677973361549 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 10:32:53.648383 10650 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 10:32:53.648496 10842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d03cd4157eee44e9a6699b03cf41b321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "033c07cc0bfd44f3b7d1969c892450cc"
I20260929 10:32:53.648710 10842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 033c07cc0bfd44f3b7d1969c892450cc [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:53.650074 10842 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 033c07cc0bfd44f3b7d1969c892450cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d03cd4157eee44e9a6699b03cf41b321 in term 1.
I20260929 10:32:53.650547 10669 leader_election.cc:302] T 00000000000000000000000000000000 P d03cd4157eee44e9a6699b03cf41b321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 033c07cc0bfd44f3b7d1969c892450cc, d03cd4157eee44e9a6699b03cf41b321; no voters:
I20260929 10:32:53.650940 10876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d03cd4157eee44e9a6699b03cf41b321 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:32:53.651198 10876 raft_consensus.cc:686] T 00000000000000000000000000000000 P d03cd4157eee44e9a6699b03cf41b321 [term 1 LEADER]: Becoming Leader. State: Replica: d03cd4157eee44e9a6699b03cf41b321, State: Running, Role: LEADER
I20260929 10:32:53.651667 10876 consensus_queue.cc:237] T 00000000000000000000000000000000 P d03cd4157eee44e9a6699b03cf41b321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03cd4157eee44e9a6699b03cf41b321" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34557 } } peers { permanent_uuid: "4fb929af0af84e16b7f497a28dd12dc9" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 37187 } } peers { permanent_uuid: "033c07cc0bfd44f3b7d1969c892450cc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40397 } }
I20260929 10:32:53.652621 10898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 033c07cc0bfd44f3b7d1969c892450cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03cd4157eee44e9a6699b03cf41b321" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34557 } } peers { permanent_uuid: "4fb929af0af84e16b7f497a28dd12dc9" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 37187 } } peers { permanent_uuid: "033c07cc0bfd44f3b7d1969c892450cc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40397 } } }
I20260929 10:32:53.652701 10793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fb929af0af84e16b7f497a28dd12dc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03cd4157eee44e9a6699b03cf41b321" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34557 } } peers { permanent_uuid: "4fb929af0af84e16b7f497a28dd12dc9" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 37187 } } peers { permanent_uuid: "033c07cc0bfd44f3b7d1969c892450cc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40397 } }
I20260929 10:32:53.652880 10898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 033c07cc0bfd44f3b7d1969c892450cc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:32:53.652910 10793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fb929af0af84e16b7f497a28dd12dc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fb929af0af84e16b7f497a28dd12dc9, State: Initialized, Role: FOLLOWER
I20260929 10:32:53.653218 10793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fb929af0af84e16b7f497a28dd12dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03cd4157eee44e9a6699b03cf41b321" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34557 } } peers { permanent_uuid: "4fb929af0af84e16b7f497a28dd12dc9" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 37187 } } peers { permanent_uuid: "033c07cc0bfd44f3b7d1969c892450cc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40397 } }
I20260929 10:32:53.654544 10793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fb929af0af84e16b7f497a28dd12dc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:32:53.654605 10767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d03cd4157eee44e9a6699b03cf41b321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4fb929af0af84e16b7f497a28dd12dc9"
I20260929 10:32:53.654836 10767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fb929af0af84e16b7f497a28dd12dc9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:53.655229 10876 sys_catalog.cc:455] T 00000000000000000000000000000000 P d03cd4157eee44e9a6699b03cf41b321 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d03cd4157eee44e9a6699b03cf41b321. Latest consensus state: current_term: 1 leader_uuid: "d03cd4157eee44e9a6699b03cf41b321" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03cd4157eee44e9a6699b03cf41b321" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34557 } } peers { permanent_uuid: "4fb929af0af84e16b7f497a28dd12dc9" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 37187 } } peers { permanent_uuid: "033c07cc0bfd44f3b7d1969c892450cc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40397 } } }
I20260929 10:32:53.656100 10767 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4fb929af0af84e16b7f497a28dd12dc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d03cd4157eee44e9a6699b03cf41b321 in term 1.
I20260929 10:32:53.656224 10876 sys_catalog.cc:458] T 00000000000000000000000000000000 P d03cd4157eee44e9a6699b03cf41b321 [sys.catalog]: This master's current role is: LEADER
W20260929 10:32:53.658764 10909 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 033c07cc0bfd44f3b7d1969c892450cc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:32:53.658890 10909 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:32:53.657262 10799 master_runner.cc:186] Error getting master registration for 127.20.92.61:37187: OK, instance_id { permanent_uuid: "4fb929af0af84e16b7f497a28dd12dc9" instance_seqno: 1790677973361549 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 10:32:53.660538 10911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb929af0af84e16b7f497a28dd12dc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03cd4157eee44e9a6699b03cf41b321" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34557 } } peers { permanent_uuid: "4fb929af0af84e16b7f497a28dd12dc9" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 37187 } } peers { permanent_uuid: "033c07cc0bfd44f3b7d1969c892450cc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40397 } } }
I20260929 10:32:53.660748 10799 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 10:32:53.660763 10911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb929af0af84e16b7f497a28dd12dc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:32:53.658811 10796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.644541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59048 (local address 127.20.92.61:37187)
0929 10:32:53.644686 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:53.644691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:53.645247 (+ 556us) server_negotiation.cc:407] Connection header received
0929 10:32:53.645439 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:53.645443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:53.645491 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:53.645558 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:53.646938 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.647855 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.649059 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.649367 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.649446 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:53.651542 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:53.651546 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:53.651549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:53.651574 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:53.654235 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:53.656909 (+ 2674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:53.656912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:53.656914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:53.656995 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:53.657606 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:53.657610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:53.657611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:53.657849 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:53.657949 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:53.658545 (+ 596us) server_negotiation.cc:299] Negotiation successful
0929 10:32:53.658662 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 10:32:53.661715 10913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:32:53.662737 10913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260929 10:32:53.667768 10925 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4fb929af0af84e16b7f497a28dd12dc9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:32:53.667852 10925 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:32:53.669591 10842 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 033c07cc0bfd44f3b7d1969c892450cc [term 1 FOLLOWER]: Refusing update from remote peer d03cd4157eee44e9a6699b03cf41b321: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:32:53.670342 10876 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d03cd4157eee44e9a6699b03cf41b321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "033c07cc0bfd44f3b7d1969c892450cc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:32:53.672612 10871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.645238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58490 (local address 127.20.92.60:40397)
0929 10:32:53.645558 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:53.645564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:53.656848 (+ 11284us) server_negotiation.cc:407] Connection header received
0929 10:32:53.656890 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:53.656892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:53.656942 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:53.657026 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:53.661612 (+ 4586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.662387 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.663981 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.664326 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.664427 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:53.666126 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:53.666130 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:53.666133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:53.666158 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:53.667973 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:53.668454 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:53.668457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:53.668459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:53.668565 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:53.668866 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:53.668869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:53.668871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:53.669086 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:53.669213 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:53.672322 (+ 3109us) server_negotiation.cc:299] Negotiation successful
0929 10:32:53.672443 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249}
I20260929 10:32:53.674588 10898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 033c07cc0bfd44f3b7d1969c892450cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader d03cd4157eee44e9a6699b03cf41b321. Latest consensus state: current_term: 1 leader_uuid: "d03cd4157eee44e9a6699b03cf41b321" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03cd4157eee44e9a6699b03cf41b321" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34557 } } peers { permanent_uuid: "4fb929af0af84e16b7f497a28dd12dc9" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 37187 } } peers { permanent_uuid: "033c07cc0bfd44f3b7d1969c892450cc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40397 } } }
I20260929 10:32:53.674705 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:53.645713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43900 (local address 127.20.92.62:34557)
0929 10:32:53.645850 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:53.645854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:53.651998 (+ 6144us) server_negotiation.cc:407] Connection header received
0929 10:32:53.652043 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:53.652047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:53.652096 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:53.652177 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:53.655059 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.655799 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.659183 (+ 3384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:53.659509 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:53.659602 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:53.670062 (+ 10460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:53.670066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:53.670068 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:53.670097 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:53.671792 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:53.673126 (+ 1334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:53.673130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:53.673132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:53.673224 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:53.673640 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:53.673644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:53.673646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:53.673917 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:53.674046 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:53.674398 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 10:32:53.674522 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 10:32:53.674998 10898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 033c07cc0bfd44f3b7d1969c892450cc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:32:53.676221 10767 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4fb929af0af84e16b7f497a28dd12dc9 [term 1 FOLLOWER]: Refusing update from remote peer d03cd4157eee44e9a6699b03cf41b321: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:32:53.676848 10876 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d03cd4157eee44e9a6699b03cf41b321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fb929af0af84e16b7f497a28dd12dc9" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 37187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:32:53.680370 10910 sys_catalog.cc:455] T 00000000000000000000000000000000 P d03cd4157eee44e9a6699b03cf41b321 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d03cd4157eee44e9a6699b03cf41b321" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03cd4157eee44e9a6699b03cf41b321" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34557 } } peers { permanent_uuid: "4fb929af0af84e16b7f497a28dd12dc9" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 37187 } } peers { permanent_uuid: "033c07cc0bfd44f3b7d1969c892450cc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40397 } } }
I20260929 10:32:53.680502 10910 sys_catalog.cc:458] T 00000000000000000000000000000000 P d03cd4157eee44e9a6699b03cf41b321 [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:53.680684 10911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb929af0af84e16b7f497a28dd12dc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d03cd4157eee44e9a6699b03cf41b321. Latest consensus state: current_term: 1 leader_uuid: "d03cd4157eee44e9a6699b03cf41b321" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03cd4157eee44e9a6699b03cf41b321" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34557 } } peers { permanent_uuid: "4fb929af0af84e16b7f497a28dd12dc9" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 37187 } } peers { permanent_uuid: "033c07cc0bfd44f3b7d1969c892450cc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40397 } } }
I20260929 10:32:53.680806 10911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb929af0af84e16b7f497a28dd12dc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:32:53.681207 10926 mvcc.cc:205] Tried to move back new op lower bound from 7334616980142415872 to 7334616980081295360. Current Snapshot: MvccSnapshot[applied={T|T < 7334616980142415872}]
I20260929 10:32:53.681663 10898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 033c07cc0bfd44f3b7d1969c892450cc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d03cd4157eee44e9a6699b03cf41b321" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03cd4157eee44e9a6699b03cf41b321" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34557 } } peers { permanent_uuid: "4fb929af0af84e16b7f497a28dd12dc9" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 37187 } } peers { permanent_uuid: "033c07cc0bfd44f3b7d1969c892450cc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40397 } } }
I20260929 10:32:53.681809 10898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 033c07cc0bfd44f3b7d1969c892450cc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:32:53.682581 10928 mvcc.cc:205] Tried to move back new op lower bound from 7334616980142415872 to 7334616980081295360. Current Snapshot: MvccSnapshot[applied={T|T < 7334616980142415872}]
I20260929 10:32:53.689167 10913 catalog_manager.cc:1347] Generated new cluster ID: b281ecc0e187492d88bafbcb82f8af14
I20260929 10:32:53.689249 10913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:32:53.706929 10876 sys_catalog.cc:455] T 00000000000000000000000000000000 P d03cd4157eee44e9a6699b03cf41b321 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d03cd4157eee44e9a6699b03cf41b321" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03cd4157eee44e9a6699b03cf41b321" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34557 } } peers { permanent_uuid: "4fb929af0af84e16b7f497a28dd12dc9" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 37187 } } peers { permanent_uuid: "033c07cc0bfd44f3b7d1969c892450cc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40397 } } }
I20260929 10:32:53.707074 10876 sys_catalog.cc:458] T 00000000000000000000000000000000 P d03cd4157eee44e9a6699b03cf41b321 [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:53.708166 10911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb929af0af84e16b7f497a28dd12dc9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d03cd4157eee44e9a6699b03cf41b321" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03cd4157eee44e9a6699b03cf41b321" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 34557 } } peers { permanent_uuid: "4fb929af0af84e16b7f497a28dd12dc9" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 37187 } } peers { permanent_uuid: "033c07cc0bfd44f3b7d1969c892450cc" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40397 } } }
I20260929 10:32:53.708292 10911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb929af0af84e16b7f497a28dd12dc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:32:53.717914 10930 mvcc.cc:205] Tried to move back new op lower bound from 7334616980142415872 to 7334616980081295360. Current Snapshot: MvccSnapshot[applied={T|T < 7334616980142415872}]
I20260929 10:32:53.731384 10913 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:32:53.732786 10913 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:32:53.752511 10913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d03cd4157eee44e9a6699b03cf41b321: Generated new TSK 0
I20260929 10:32:53.753592 10913 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 10:32:53.784416 10796 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:32:53.780650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59050 (local address 127.20.92.61:37187)
0929 10:32:53.780788 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:53.780793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:53.782950 (+ 2157us) server_negotiation.cc:407] Connection header received
0929 10:32:53.783382 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:53.783386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:53.783442 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:53.783520 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:53.784226 (+ 706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59050: BlockingRecv error: recv got EOF from 127.0.0.1:59050 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 10:32:53.786072 10871 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:32:53.781424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58506 (local address 127.20.92.60:40397)
0929 10:32:53.781573 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:53.781578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:53.783293 (+ 1715us) server_negotiation.cc:407] Connection header received
0929 10:32:53.785216 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:53.785220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:53.785273 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:53.785327 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:53.785937 (+ 610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58506: BlockingRecv error: recv got EOF from 127.0.0.1:58506 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 10:32:53.787436 10721 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:32:53.782085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43906 (local address 127.20.92.62:34557)
0929 10:32:53.782205 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:53.782210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:53.785017 (+ 2807us) server_negotiation.cc:407] Connection header received
0929 10:32:53.786595 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:53.786599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:53.786653 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:53.786707 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:53.787310 (+ 603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43906: BlockingRecv error: recv got EOF from 127.0.0.1:43906 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 10:32:53.792651 10796 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:32:53.791038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59054 (local address 127.20.92.61:37187)
0929 10:32:53.791151 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:53.791156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:53.791610 (+ 454us) server_negotiation.cc:407] Connection header received
0929 10:32:53.791843 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:53.791846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:53.791924 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:53.791980 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:53.792549 (+ 569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59054: BlockingRecv error: recv got EOF from 127.0.0.1:59054 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 10:32:53.798048 10796 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:32:53.796402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59066 (local address 127.20.92.61:37187)
0929 10:32:53.796521 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:53.796525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:53.797071 (+ 546us) server_negotiation.cc:407] Connection header received
0929 10:32:53.797262 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:53.797265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:53.797316 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:53.797369 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:53.797945 (+ 576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59066: BlockingRecv error: recv got EOF from 127.0.0.1:59066 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 10:32:53.805022 10796 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:32:53.802705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59068 (local address 127.20.92.61:37187)
0929 10:32:53.802855 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:53.802860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:53.803691 (+ 831us) server_negotiation.cc:407] Connection header received
0929 10:32:53.803984 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:53.803988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:53.804070 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:53.804153 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:53.804888 (+ 735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59068: BlockingRecv error: recv got EOF from 127.0.0.1:59068 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 10:32:53.819458 10796 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:32:53.814277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59070 (local address 127.20.92.61:37187)
0929 10:32:53.814432 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:53.814439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:53.818350 (+ 3911us) server_negotiation.cc:407] Connection header received
0929 10:32:53.818614 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:53.818617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:53.818676 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:53.818734 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:53.819343 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59070: BlockingRecv error: recv got EOF from 127.0.0.1:59070 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 10:32:53.832238 10796 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:32:53.830259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59082 (local address 127.20.92.61:37187)
0929 10:32:53.830390 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:53.830396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:53.831127 (+ 731us) server_negotiation.cc:407] Connection header received
0929 10:32:53.831363 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:53.831368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:53.831428 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:53.831506 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:53.832122 (+ 616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59082: BlockingRecv error: recv got EOF from 127.0.0.1:59082 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 10:32:53.853286 10796 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:32:53.851290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59096 (local address 127.20.92.61:37187)
0929 10:32:53.851440 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:53.851445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:53.852100 (+ 655us) server_negotiation.cc:407] Connection header received
0929 10:32:53.852345 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:53.852349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:53.852412 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:53.852493 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:53.853128 (+ 635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59096: BlockingRecv error: recv got EOF from 127.0.0.1:59096 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 10:32:53.889256 10796 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:32:53.887899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59108 (local address 127.20.92.61:37187)
0929 10:32:53.888036 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:53.888040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:53.888295 (+ 255us) server_negotiation.cc:407] Connection header received
0929 10:32:53.888511 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:53.888515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:53.888573 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:53.888671 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:53.889136 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59108: BlockingRecv error: recv got EOF from 127.0.0.1:59108 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 10:32:53.956557 10796 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:32:53.955417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59120 (local address 127.20.92.61:37187)
0929 10:32:53.955569 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:53.955573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:53.955684 (+ 111us) server_negotiation.cc:407] Connection header received
0929 10:32:53.955887 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:53.955890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:53.955941 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:53.956026 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:53.956417 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59120: BlockingRecv error: recv got EOF from 127.0.0.1:59120 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 10:32:54.088063 10796 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:32:54.086753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59126 (local address 127.20.92.61:37187)
0929 10:32:54.086911 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:54.086917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:54.087099 (+ 182us) server_negotiation.cc:407] Connection header received
0929 10:32:54.087295 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:54.087298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:54.087348 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:54.087444 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:54.087912 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59126: BlockingRecv error: recv got EOF from 127.0.0.1:59126 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 10:32:54.347642 10796 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:32:54.346340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59142 (local address 127.20.92.61:37187)
0929 10:32:54.346480 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:54.346485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:54.346680 (+ 195us) server_negotiation.cc:407] Connection header received
0929 10:32:54.346886 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:54.346889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:54.346937 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:54.347036 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:54.347503 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59142: BlockingRecv error: recv got EOF from 127.0.0.1:59142 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 10:32:54.660202 10909 catalog_manager.cc:1259] Loaded cluster ID: b281ecc0e187492d88bafbcb82f8af14
I20260929 10:32:54.660290 10909 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 033c07cc0bfd44f3b7d1969c892450cc: loading cluster ID for follower catalog manager: success
I20260929 10:32:54.661984 10909 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 033c07cc0bfd44f3b7d1969c892450cc: acquiring CA information for follower catalog manager: success
I20260929 10:32:54.663182 10909 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 033c07cc0bfd44f3b7d1969c892450cc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 10:32:54.669034 10925 catalog_manager.cc:1259] Loaded cluster ID: b281ecc0e187492d88bafbcb82f8af14
I20260929 10:32:54.669114 10925 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4fb929af0af84e16b7f497a28dd12dc9: loading cluster ID for follower catalog manager: success
I20260929 10:32:54.670763 10925 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4fb929af0af84e16b7f497a28dd12dc9: acquiring CA information for follower catalog manager: success
I20260929 10:32:54.671875 10925 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4fb929af0af84e16b7f497a28dd12dc9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 10:32:54.865985 10998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:54.862027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59148 (local address 127.20.92.61:37187)
0929 10:32:54.862283 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:54.862287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:54.862513 (+ 226us) server_negotiation.cc:407] Connection header received
0929 10:32:54.862577 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:54.862579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:54.862703 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:54.862805 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:32:54.863086 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:54.863632 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:54.864453 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:54.864708 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:54.864786 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:54.865074 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:32:54.865518 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:32:54.865628 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:54.865720 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 10:32:54.865842 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":77,"threads_started":1}
I20260929 10:32:54.872857 11005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:54.869257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58522 (local address 127.20.92.60:40397)
0929 10:32:54.869542 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:54.869546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:54.869560 (+ 14us) server_negotiation.cc:407] Connection header received
0929 10:32:54.869636 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:54.869638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:54.869747 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:54.869876 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:32:54.870125 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:54.870664 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:54.871488 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:54.871707 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:54.871790 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:54.872079 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:32:54.872533 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:32:54.872626 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:54.872682 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 10:32:54.872733 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":106,"threads_started":1}
I20260929 10:32:54.880002 10998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:54.876019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59164 (local address 127.20.92.61:37187)
0929 10:32:54.876178 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:54.876184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:54.876288 (+ 104us) server_negotiation.cc:407] Connection header received
0929 10:32:54.876371 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:54.876373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:54.876435 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:54.876514 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:32:54.876864 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:54.877581 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:54.878779 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:54.879127 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:54.879244 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:54.879526 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:32:54.879612 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:32:54.879709 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:54.879809 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 10:32:54.879868 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 10:32:54.880524 11005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:54.876363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58526 (local address 127.20.92.60:40397)
0929 10:32:54.876505 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:54.876511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:54.876822 (+ 311us) server_negotiation.cc:407] Connection header received
0929 10:32:54.877024 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:54.877027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:54.877085 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:54.877133 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:32:54.877496 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:54.878217 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:54.879251 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:54.879568 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:54.879651 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:54.880023 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:32:54.880134 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:32:54.880210 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:54.880348 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 10:32:54.880403 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 10:32:54.880795 11011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:54.875900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43918 (local address 127.20.92.62:34557)
0929 10:32:54.876512 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:54.876517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:54.876531 (+ 14us) server_negotiation.cc:407] Connection header received
0929 10:32:54.876641 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:54.876645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:54.876824 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:54.876969 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:32:54.877140 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:54.877971 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:54.879031 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:54.879337 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:54.879450 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:54.879777 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:32:54.880150 (+ 373us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:32:54.880523 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:54.880558 (+ 35us) server_negotiation.cc:299] Negotiation successful
0929 10:32:54.880656 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":84,"threads_started":1}
I20260929 10:32:54.884405 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 10650
I20260929 10:32:54.890981 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 10725
I20260929 10:32:54.897009 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 10799
2026-09-29T10:32:54Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3138 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:32:54 dist-test-slave-9cj5 krb5kdc[11016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:32:54 dist-test-slave-9cj5 krb5kdc[11016](info): set up 2 sockets
Sep 29 10:32:54 dist-test-slave-9cj5 krb5kdc[11016](info): commencing operation
krb5kdc: starting...
W20260929 10:32:55.946879 20848 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/krb5kdc/test-user.keytab.
Sep 29 10:32:56 dist-test-slave-9cj5 krb5kdc[11016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:32:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:32:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
Entry for principal kudu/127.20.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab.
I20260929 10:32:56.072141 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:45813
--webserver_interface=127.20.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:39369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:45813,127.20.92.61:35155,127.20.92.60:40351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:32:56.181457 11029 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 10:32:56.181804 11029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:56.181846 11029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:56.181890 11029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:32:56.185914 11029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:32:56.185971 11029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:56.185990 11029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:56.186007 11029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:32:56.186023 11029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:32:56.191293 11029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:39369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:45813,127.20.92.61:35155,127.20.92.60:40351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.62.keytab
--principal=kudu/127.20.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.62:45813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:56.192303 11029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:56.193248 11029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:56.201647 11037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:56.201831 11029 server_base.cc:1034] running on GCE node
W20260929 10:32:56.201689 11034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:56.201717 11035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:56.202540 11029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:56.203560 11029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:56.204754 11029 hybrid_clock.cc:648] HybridClock initialized: now 1790677976204724 us; error 42 us; skew 500 ppm
Sep 29 10:32:56 dist-test-slave-9cj5 krb5kdc[11016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677976, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:56.207571 11029 init.cc:377] Logged in from keytab as kudu/127.20.92.62@KRBTEST.COM (short username kudu)
I20260929 10:32:56.208508 11029 webserver.cc:460] Webserver started at http://127.20.92.62:37893/ using document root <none> and password file <none>
I20260929 10:32:56.209061 11029 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:56.209159 11029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:56.209389 11029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:56.211458 11029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-0/data/instance:
uuid: "d3805215dcbe44cd808ee9537ae0227d"
format_stamp: "Formatted at 2026-09-29 10:32:56 on dist-test-slave-9cj5"
server_key: "a5ee88cae8ca19f48671861ef46cbfc2"
server_key_iv: "adcb96dc258e51c464f95c527137fc48"
server_key_version: "encryptionkey@0"
I20260929 10:32:56.212020 11029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-0/wal/instance:
uuid: "d3805215dcbe44cd808ee9537ae0227d"
format_stamp: "Formatted at 2026-09-29 10:32:56 on dist-test-slave-9cj5"
server_key: "a5ee88cae8ca19f48671861ef46cbfc2"
server_key_iv: "adcb96dc258e51c464f95c527137fc48"
server_key_version: "encryptionkey@0"
I20260929 10:32:56.215708 11029 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 10:32:56.218276 11044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:56.219563 11029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:32:56.219666 11029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-0/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-0/wal
uuid: "d3805215dcbe44cd808ee9537ae0227d"
format_stamp: "Formatted at 2026-09-29 10:32:56 on dist-test-slave-9cj5"
server_key: "a5ee88cae8ca19f48671861ef46cbfc2"
server_key_iv: "adcb96dc258e51c464f95c527137fc48"
server_key_version: "encryptionkey@0"
I20260929 10:32:56.219755 11029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:56.236898 11029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:56.240185 11029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:56.240339 11029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:56.247466 11029 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.62:45813
I20260929 10:32:56.247859 11096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.62:45813 every 8 connection(s)
I20260929 10:32:56.248462 11029 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-0/data/info.pb
I20260929 10:32:56.251375 11097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:56.255551 11097 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45813 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:32:56.258453 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 11029
I20260929 10:32:56.258572 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-0/wal/instance
I20260929 10:32:56.258919 20848 external_mini_cluster.cc:1428] Setting key 8fc4a2e0c2e033deac5bac34de4695e8
Sep 29 10:32:56 dist-test-slave-9cj5 krb5kdc[11016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677976, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.92.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.61@KRBTEST.COM" created.
I20260929 10:32:56.271376 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.257739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45813 (local address 127.0.0.1:32956)
0929 10:32:56.258298 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:56.258340 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:56.259402 (+ 1062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:56.260226 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:56.260230 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:56.260732 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:56.261109 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:56.261130 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.262760 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.262763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:56.263598 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:56.263607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.264621 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.264623 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:56.264732 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:56.265332 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:32:56.265347 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:56.266798 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:56.268659 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:56.268667 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:56.268675 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:56.268892 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:56.269189 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:56.269191 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:56.269192 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:56.269292 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:56.269934 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:56.269939 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:56.270061 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 10:32:56.270288 (+ 227us) client_negotiation.cc:240] Negotiation successful
0929 10:32:56.270665 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":99,"thread_start_us":110,"threads_started":1}
I20260929 10:32:56.271374 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.258671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32956 (local address 127.20.92.62:45813)
0929 10:32:56.259102 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:56.259114 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:56.259157 (+ 43us) server_negotiation.cc:407] Connection header received
0929 10:32:56.259756 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:56.259765 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:56.260007 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:56.260210 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:56.261294 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.262620 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.263726 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.264429 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.264628 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:56.266953 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:56.266960 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:56.266972 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:56.266997 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:56.268553 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:56.268989 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:56.268994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:56.268995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:56.269107 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:56.269372 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:56.269374 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:56.269375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:56.269830 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:56.269935 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:56.270329 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 10:32:56.270591 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":326,"thread_start_us":96,"threads_started":1}
I20260929 10:32:56.274058 11097 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.92.61" port: 35155 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:32:56.274816 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.274537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:35155 (local address 127.0.0.1:47006)
0929 10:32:56.274692 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:56.274764 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.61:35155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260929 10:32:56.275027 11045 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.92.61:35155: connect: Connection refused (error 111)
I20260929 10:32:56.275816 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.275618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:35155 (local address 127.0.0.1:47008)
0929 10:32:56.275722 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:56.275770 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.61:35155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 10:32:56.275977 11097 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.92.61:35155: Network error: Client connection negotiation failed: client connection to 127.20.92.61:35155: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.61.keytab.
Entry for principal kudu/127.20.92.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.61.keytab.
I20260929 10:32:56.287338 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.61:35155
--webserver_interface=127.20.92.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.61.keytab
--principal=kudu/127.20.92.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:39369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:45813,127.20.92.61:35155,127.20.92.60:40351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:32:56.325201 11097 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.92.61" port: 35155 } attempt: 1
I20260929 10:32:56.326102 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.325799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:35155 (local address 127.0.0.1:47014)
0929 10:32:56.325949 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:56.326012 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.61:35155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 10:32:56.326799 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.326591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:35155 (local address 127.0.0.1:47030)
0929 10:32:56.326696 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:56.326747 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.61:35155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 10:32:56.326930 11097 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.92.61:35155: Network error: Client connection negotiation failed: client connection to 127.20.92.61:35155: connect: Connection refused (error 111)
I20260929 10:32:56.395095 11097 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.92.61" port: 35155 } attempt: 2
W20260929 10:32:56.395059 11104 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 10:32:56.395449 11104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:56.395501 11104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:56.395561 11104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 10:32:56.395967 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.395650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:35155 (local address 127.0.0.1:47046)
0929 10:32:56.395801 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:56.395894 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.61:35155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 10:32:56.396641 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.396429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:35155 (local address 127.0.0.1:47052)
0929 10:32:56.396531 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:56.396578 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.61:35155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 10:32:56.396765 11097 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.92.61:35155: Network error: Client connection negotiation failed: client connection to 127.20.92.61:35155: connect: Connection refused (error 111)
W20260929 10:32:56.400204 11104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:32:56.400301 11104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:56.400349 11104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:56.400386 11104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:32:56.400439 11104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:32:56.406000 11104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:39369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:45813,127.20.92.61:35155,127.20.92.60:40351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.61.keytab
--principal=kudu/127.20.92.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.61:35155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.92.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:56.407060 11104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:56.408068 11104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:32:56.416558 11110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:56.416657 11109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:56.416683 11104 server_base.cc:1034] running on GCE node
W20260929 10:32:56.416755 11112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:56.417469 11104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:56.418692 11104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:56.419922 11104 hybrid_clock.cc:648] HybridClock initialized: now 1790677976419904 us; error 31 us; skew 500 ppm
Sep 29 10:32:56 dist-test-slave-9cj5 krb5kdc[11016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677976, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:56.422982 11104 init.cc:377] Logged in from keytab as kudu/127.20.92.61@KRBTEST.COM (short username kudu)
I20260929 10:32:56.423893 11104 webserver.cc:460] Webserver started at http://127.20.92.61:41023/ using document root <none> and password file <none>
I20260929 10:32:56.424439 11104 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:56.424575 11104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:56.424806 11104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:56.426851 11104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-1/data/instance:
uuid: "bbe3672e8af943d384ca88a908c02724"
format_stamp: "Formatted at 2026-09-29 10:32:56 on dist-test-slave-9cj5"
server_key: "a2be721006c85b7312d1dd7a55027d00"
server_key_iv: "b55978c157e77d349da07ef1fe42d735"
server_key_version: "encryptionkey@0"
I20260929 10:32:56.427419 11104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-1/wal/instance:
uuid: "bbe3672e8af943d384ca88a908c02724"
format_stamp: "Formatted at 2026-09-29 10:32:56 on dist-test-slave-9cj5"
server_key: "a2be721006c85b7312d1dd7a55027d00"
server_key_iv: "b55978c157e77d349da07ef1fe42d735"
server_key_version: "encryptionkey@0"
I20260929 10:32:56.431139 11104 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:32:56.433512 11119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:56.434866 11104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:32:56.435017 11104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-1/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-1/wal
uuid: "bbe3672e8af943d384ca88a908c02724"
format_stamp: "Formatted at 2026-09-29 10:32:56 on dist-test-slave-9cj5"
server_key: "a2be721006c85b7312d1dd7a55027d00"
server_key_iv: "b55978c157e77d349da07ef1fe42d735"
server_key_version: "encryptionkey@0"
I20260929 10:32:56.435135 11104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:56.460493 11104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:56.463354 11104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:56.463585 11104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:56.470176 11104 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.61:35155
I20260929 10:32:56.470451 11171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.61:35155 every 8 connection(s)
I20260929 10:32:56.471127 11104 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-1/data/info.pb
I20260929 10:32:56.473714 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 11104
I20260929 10:32:56.473870 11172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:56.473889 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-1/wal/instance
I20260929 10:32:56.474188 20848 external_mini_cluster.cc:1428] Setting key 8894583a2ce2715938fbf7507f28572a
I20260929 10:32:56.478292 11172 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45813 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.92.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.92.60@KRBTEST.COM" created.
I20260929 10:32:56.487922 11097 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.92.61" port: 35155 } attempt: 3
Sep 29 10:32:56 dist-test-slave-9cj5 krb5kdc[11016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677976, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.61@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:32:56.493151 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.480206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32958 (local address 127.20.92.62:45813)
0929 10:32:56.480340 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:56.480344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:56.481355 (+ 1011us) server_negotiation.cc:407] Connection header received
0929 10:32:56.482392 (+ 1037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:56.482395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:56.482458 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:56.482556 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:56.483663 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.484191 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.485369 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.485616 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.485729 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:56.488965 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:56.488969 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:56.488972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:56.489001 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:56.490951 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:56.491551 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:56.491558 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:56.491561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:56.491654 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:56.492030 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:56.492034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:56.492036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:56.492224 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:56.492335 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:56.492877 (+ 542us) server_negotiation.cc:299] Negotiation successful
0929 10:32:56.493010 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 10:32:56.494642 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.480710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45813 (local address 127.0.0.1:32958)
0929 10:32:56.481206 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:56.481247 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:56.482184 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:56.482723 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:56.482730 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:56.483059 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:56.483514 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:56.483525 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.484318 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.484323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:56.485240 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:56.485250 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.485793 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.485798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:56.485962 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:56.486621 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:32:56.486648 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:56.488770 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:56.491059 (+ 2289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:56.491065 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:56.491078 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:56.491415 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:56.491760 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:56.491763 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:56.491765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:56.491951 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:56.492334 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:56.492341 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:56.492597 (+ 256us) client_negotiation.cc:769] Sending connection context
0929 10:32:56.493383 (+ 786us) client_negotiation.cc:240] Negotiation successful
0929 10:32:56.493736 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":135,"threads_started":1}
I20260929 10:32:56.496408 11172 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.92.61" port: 35155 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 10:32:56 dist-test-slave-9cj5 krb5kdc[11016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677976, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for kudu/127.20.92.61@KRBTEST.COM
I20260929 10:32:56.501545 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.488405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:35155 (local address 127.0.0.1:47058)
0929 10:32:56.488623 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:56.488639 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:56.488721 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:56.489507 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:56.489510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:56.489713 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:56.489866 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:56.489872 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.491020 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.491027 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:56.491849 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:56.491860 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.492721 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.492724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:56.492849 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:56.493498 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:32:56.493511 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:56.497369 (+ 3858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:56.499769 (+ 2400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:56.499776 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:56.499780 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:56.500050 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:56.500328 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:56.500331 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:56.500333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:56.500437 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:56.500973 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:56.500976 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:56.501077 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 10:32:56.501196 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 10:32:56.501334 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.92.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.60.keytab.
Entry for principal kudu/127.20.92.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.60.keytab.
I20260929 10:32:56.502002 11177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.488490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47058 (local address 127.20.92.61:35155)
0929 10:32:56.488968 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:56.488980 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:56.489004 (+ 24us) server_negotiation.cc:407] Connection header received
0929 10:32:56.489048 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:56.489055 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:56.489374 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:56.489528 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:56.490192 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.490895 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.491973 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.492592 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.492709 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:56.497545 (+ 4836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:56.497554 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:56.497564 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:56.497597 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:56.499637 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:56.500158 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:56.500162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:56.500163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:56.500237 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:56.500532 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:56.500535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:56.500537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:56.500816 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:56.500975 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:56.501285 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 10:32:56.501519 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"spinlock_wait_cycles":2432,"thread_start_us":87,"threads_started":1}
Sep 29 10:32:56 dist-test-slave-9cj5 krb5kdc[11016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677976, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.61@KRBTEST.COM for kudu/127.20.92.61@KRBTEST.COM
I20260929 10:32:56.503252 20848 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
/tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.60:40351
--webserver_interface=127.20.92.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.60.keytab
--principal=kudu/127.20.92.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.92.20:39369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.92.62:45813,127.20.92.61:35155,127.20.92.60:40351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:32:56.503903 11097 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40351 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:32:56.504694 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.504363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:40351 (local address 127.0.0.1:44762)
0929 10:32:56.504527 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:56.504599 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 10:32:56.505501 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.505175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:40351 (local address 127.0.0.1:44770)
0929 10:32:56.505358 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:56.505420 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260929 10:32:56.505861 11097 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.92.60:40351: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40351: connect: Connection refused (error 111)
I20260929 10:32:56.508324 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.497013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:35155 (local address 127.0.0.1:47074)
0929 10:32:56.497139 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:56.497154 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:56.497248 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:56.497679 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:56.497684 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:56.497897 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:56.498033 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:56.498039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.499153 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.499156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:56.499936 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:56.499943 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.500659 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.500661 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:56.500727 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:56.501156 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:32:56.501176 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:56.505025 (+ 3849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:56.506863 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:56.506870 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:56.506872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:56.507066 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:56.507358 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:56.507361 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:56.507363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:56.507442 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:56.507801 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:56.507804 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:56.507883 (+ 79us) client_negotiation.cc:769] Sending connection context
0929 10:32:56.508015 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 10:32:56.508156 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 10:32:56.508371 11178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.497108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47074 (local address 127.20.92.61:35155)
0929 10:32:56.497450 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:56.497462 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:56.497475 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:32:56.497511 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:56.497513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:56.497557 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:56.497624 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:56.498153 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.499041 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.500021 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.500374 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.500510 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:56.505170 (+ 4660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:56.505173 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:56.505175 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:56.505194 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:56.506754 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:56.507181 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:56.507184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:56.507186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:56.507262 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:56.507528 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:56.507533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:56.507535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:56.507706 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:56.507814 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:56.508076 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 10:32:56.508213 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":258,"spinlock_wait_cycles":3968,"thread_start_us":108,"threads_started":1}
I20260929 10:32:56.509181 11172 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40351 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:32:56.510002 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.509646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:40351 (local address 127.0.0.1:44778)
0929 10:32:56.509821 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:56.509914 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 10:32:56.510262 11123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40351: connect: Connection refused (error 111)
I20260929 10:32:56.511327 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.511045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:40351 (local address 127.0.0.1:44792)
0929 10:32:56.511181 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:56.511246 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 10:32:56.511485 11172 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.92.60:40351: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40351: connect: Connection refused (error 111)
I20260929 10:32:56.537061 11097 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40351 } attempt: 1
I20260929 10:32:56.537974 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.537638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:40351 (local address 127.0.0.1:44800)
0929 10:32:56.537808 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:56.537879 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 10:32:56.538738 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.538491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:40351 (local address 127.0.0.1:44808)
0929 10:32:56.538623 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:56.538678 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 10:32:56.538901 11097 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.92.60:40351: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40351: connect: Connection refused (error 111)
I20260929 10:32:56.560789 11172 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40351 } attempt: 1
I20260929 10:32:56.561659 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.561321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:40351 (local address 127.0.0.1:44814)
0929 10:32:56.561488 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:56.561558 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 10:32:56.562378 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.562160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:40351 (local address 127.0.0.1:44820)
0929 10:32:56.562267 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:56.562317 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 10:32:56.562505 11172 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.92.60:40351: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40351: connect: Connection refused (error 111)
I20260929 10:32:56.614070 11097 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40351 } attempt: 2
W20260929 10:32:56.614650 11179 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
I20260929 10:32:56.614917 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.614590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:40351 (local address 127.0.0.1:44828)
0929 10:32:56.614762 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:56.614827 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260929 10:32:56.614980 11179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:32:56.615095 11179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:32:56.615187 11179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 10:32:56.615662 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.615428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:40351 (local address 127.0.0.1:44840)
0929 10:32:56.615543 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:56.615602 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 10:32:56.615801 11097 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.92.60:40351: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40351: connect: Connection refused (error 111)
W20260929 10:32:56.619539 11179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:32:56.619589 11179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:32:56.619608 11179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:32:56.619650 11179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:32:56.619683 11179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:32:56.625201 11179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.92.20:39369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.92.62:45813,127.20.92.61:35155,127.20.92.60:40351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/krb5kdc/kudu_127.20.92.60.keytab
--principal=kudu/127.20.92.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.92.60:40351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.92.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:32:56.626320 11179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:32:56.627354 11179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:32:56.630658 11172 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40351 } attempt: 2
I20260929 10:32:56.631460 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.631146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:40351 (local address 127.0.0.1:44852)
0929 10:32:56.631307 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:56.631373 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 10:32:56.632290 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.632051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:40351 (local address 127.0.0.1:44856)
0929 10:32:56.632186 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:56.632238 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 10:32:56.632478 11172 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.92.60:40351: Network error: Client connection negotiation failed: client connection to 127.20.92.60:40351: connect: Connection refused (error 111)
W20260929 10:32:56.636265 11185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:32:56.636269 11186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:56.636427 11179 server_base.cc:1034] running on GCE node
W20260929 10:32:56.636276 11188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:32:56.637063 11179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:32:56.638212 11179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:32:56.639371 11179 hybrid_clock.cc:648] HybridClock initialized: now 1790677976639333 us; error 50 us; skew 500 ppm
Sep 29 10:32:56 dist-test-slave-9cj5 krb5kdc[11016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677976, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:32:56.642164 11179 init.cc:377] Logged in from keytab as kudu/127.20.92.60@KRBTEST.COM (short username kudu)
I20260929 10:32:56.643009 11179 webserver.cc:460] Webserver started at http://127.20.92.60:33139/ using document root <none> and password file <none>
I20260929 10:32:56.643565 11179 fs_manager.cc:359] Metadata directory not provided
I20260929 10:32:56.643654 11179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:32:56.643888 11179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:32:56.645936 11179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-2/data/instance:
uuid: "92fc0b3c06fa47adbbbbb65742e88150"
format_stamp: "Formatted at 2026-09-29 10:32:56 on dist-test-slave-9cj5"
server_key: "fdb2439bed446094a4e65c50cca4dc07"
server_key_iv: "f5681bb49a0ca67ef12b81e9cc5d2af0"
server_key_version: "encryptionkey@0"
I20260929 10:32:56.646519 11179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-2/wal/instance:
uuid: "92fc0b3c06fa47adbbbbb65742e88150"
format_stamp: "Formatted at 2026-09-29 10:32:56 on dist-test-slave-9cj5"
server_key: "fdb2439bed446094a4e65c50cca4dc07"
server_key_iv: "f5681bb49a0ca67ef12b81e9cc5d2af0"
server_key_version: "encryptionkey@0"
I20260929 10:32:56.650171 11179 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:32:56.652472 11195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:32:56.653718 11179 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:32:56.653936 11179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-2/data,/tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-2/wal
uuid: "92fc0b3c06fa47adbbbbb65742e88150"
format_stamp: "Formatted at 2026-09-29 10:32:56 on dist-test-slave-9cj5"
server_key: "fdb2439bed446094a4e65c50cca4dc07"
server_key_iv: "f5681bb49a0ca67ef12b81e9cc5d2af0"
server_key_version: "encryptionkey@0"
I20260929 10:32:56.654050 11179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:32:56.665632 11179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:32:56.669070 11179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:32:56.669309 11179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:32:56.676679 11179 rpc_server.cc:307] RPC server started. Bound to: 127.20.92.60:40351
I20260929 10:32:56.676977 11247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.92.60:40351 every 8 connection(s)
I20260929 10:32:56.677774 11179 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-2/data/info.pb
I20260929 10:32:56.680017 20848 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu as pid 11179
I20260929 10:32:56.680212 20848 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Q3Hzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677679287822-20848-0/minicluster-data/master-2/wal/instance
I20260929 10:32:56.680460 11248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:32:56.680634 20848 external_mini_cluster.cc:1428] Setting key d79869b1c76e4abe8ecc767ae68ef62d
I20260929 10:32:56.684867 11248 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45813 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 10:32:56 dist-test-slave-9cj5 krb5kdc[11016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.60@KRBTEST.COM
Sep 29 10:32:56 dist-test-slave-9cj5 krb5kdc[11016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677976, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.60@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:32:56.696082 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.686616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32962 (local address 127.20.92.62:45813)
0929 10:32:56.686745 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:56.686750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:56.687036 (+ 286us) server_negotiation.cc:407] Connection header received
0929 10:32:56.687179 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:56.687185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:56.687254 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:56.687317 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:56.688113 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.688858 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.689898 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.690164 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.690269 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:56.692617 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:56.692621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:56.692624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:56.692656 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:56.694177 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:56.694743 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:56.694749 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:56.694753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:56.694860 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:56.695261 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:56.695264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:56.695265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:56.695449 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:56.695535 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:56.695872 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 10:32:56.695970 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 10:32:56.696360 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.682116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44868 (local address 127.20.92.60:40351)
0929 10:32:56.682573 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:56.682588 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:56.682622 (+ 34us) server_negotiation.cc:407] Connection header received
0929 10:32:56.683538 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:56.683555 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:56.683879 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:56.684232 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:56.685045 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.686156 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.687230 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.687928 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.688103 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:56.690884 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:56.690893 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:56.690909 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:56.690947 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:56.692744 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:56.693229 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:56.693233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:56.693238 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:56.693340 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:56.693681 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:56.693683 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:56.693684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:56.694128 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:56.694364 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:56.695102 (+ 738us) server_negotiation.cc:299] Negotiation successful
0929 10:32:56.695384 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":124,"threads_started":1}
I20260929 10:32:56.696360 11252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.686549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.62:45813 (local address 127.0.0.1:32962)
0929 10:32:56.686902 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:56.686949 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:56.687067 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:56.687390 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:56.687395 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:56.687828 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:56.687989 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:56.687998 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.688990 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.688992 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:56.689742 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:56.689782 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.690319 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.690326 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:56.690443 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:56.691125 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:32:56.691137 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:56.692474 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:56.694317 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:56.694326 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:56.694332 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:56.694626 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:56.694978 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:56.694981 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:56.694983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:56.695136 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:56.695560 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:56.695566 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:56.695675 (+ 109us) client_negotiation.cc:769] Sending connection context
0929 10:32:56.695820 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 10:32:56.695981 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":136,"threads_started":1}
I20260929 10:32:56.697641 11248 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.92.61" port: 35155 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 10:32:56 dist-test-slave-9cj5 krb5kdc[11016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677976, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.60@KRBTEST.COM for kudu/127.20.92.61@KRBTEST.COM
Sep 29 10:32:56 dist-test-slave-9cj5 krb5kdc[11016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.62@KRBTEST.COM
I20260929 10:32:56.706638 11252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.698144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.61:35155 (local address 127.0.0.1:47082)
0929 10:32:56.698297 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:56.698313 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:56.698384 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:56.698623 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:56.698626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:56.698806 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:56.698921 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:56.698928 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.699804 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.699808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:56.700364 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:56.700372 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.700834 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.700837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:56.700931 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:56.701430 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:32:56.701441 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:56.702995 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:56.705154 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:56.705157 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:56.705159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:56.705327 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:56.705632 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:56.705637 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:56.705639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:56.705721 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:56.706149 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:56.706153 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:56.706243 (+ 90us) client_negotiation.cc:769] Sending connection context
0929 10:32:56.706345 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 10:32:56.706473 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 10:32:56.706697 11178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.698253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47082 (local address 127.20.92.61:35155)
0929 10:32:56.698400 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:56.698406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:56.698423 (+ 17us) server_negotiation.cc:407] Connection header received
0929 10:32:56.698466 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:56.698469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:56.698509 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:56.698614 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:56.699055 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.699692 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.700507 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.700726 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.700806 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:56.703157 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:56.703164 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:56.703167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:56.703201 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:56.704866 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:56.705440 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:56.705443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:56.705445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:56.705531 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:56.705859 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:56.705862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:56.705863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:56.706040 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:56.706148 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:56.706409 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 10:32:56.706524 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 10:32:56.707515 11248 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40351 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:32:56.707803 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.698882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32978 (local address 127.20.92.62:45813)
0929 10:32:56.699009 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:56.699015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:56.699085 (+ 70us) server_negotiation.cc:407] Connection header received
0929 10:32:56.699198 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:56.699201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:56.699244 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:56.699318 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:56.699907 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.700359 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.701321 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.701576 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.701659 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:56.704442 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:56.704446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:56.704449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:56.704472 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:56.706032 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:56.706488 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:56.706494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:56.706496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:56.706585 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:56.706894 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:56.706899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:56.706900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:56.707096 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:56.707375 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:56.707524 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 10:32:56.707652 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Sep 29 10:32:56 dist-test-slave-9cj5 krb5kdc[11016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677976, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.60@KRBTEST.COM for kudu/127.20.92.60@KRBTEST.COM
Sep 29 10:32:56 dist-test-slave-9cj5 krb5kdc[11016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.92.61@KRBTEST.COM
I20260929 10:32:56.714978 11097 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40351 } attempt: 3
I20260929 10:32:56.717608 11252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.708023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:40351 (local address 127.0.0.1:44874)
0929 10:32:56.708247 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:56.708260 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:56.708324 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:56.708582 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:56.708599 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:56.708774 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:56.708862 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:56.708873 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.710495 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.710500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:56.711044 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:56.711053 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.711502 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.711506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:56.711593 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:56.712068 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:32:56.712079 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:56.713349 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:56.715735 (+ 2386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:56.715739 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:56.715742 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:56.716215 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:56.716518 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:56.716522 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:56.716524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:56.716620 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:56.717082 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:56.717084 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:56.717144 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 10:32:56.717272 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 10:32:56.717413 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260929 10:32:56.718468 11178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.709302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47096 (local address 127.20.92.61:35155)
0929 10:32:56.709408 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:56.709412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:56.709478 (+ 66us) server_negotiation.cc:407] Connection header received
0929 10:32:56.709634 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:56.709636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:56.709672 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:56.709739 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:56.710425 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.710929 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.711841 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.712188 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.712271 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:56.714858 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:56.714863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:56.714867 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:56.714893 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:56.716550 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:56.717004 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:56.717008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:56.717010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:56.717103 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:56.717459 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:56.717463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:56.717465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:56.717679 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:56.717904 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:56.718214 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 10:32:56.718324 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 10:32:56.717641 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.708260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44874 (local address 127.20.92.60:40351)
0929 10:32:56.708381 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:56.708387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:56.708402 (+ 15us) server_negotiation.cc:407] Connection header received
0929 10:32:56.708439 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:56.708442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:56.708488 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:56.708579 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:56.708972 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.710418 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.711158 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.711398 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.711479 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:56.713518 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:56.713522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:56.713525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:56.713550 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:56.715315 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:56.716314 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:56.716320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:56.716322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:56.716423 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:56.716700 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:56.716703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:56.716705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:56.716957 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:56.717044 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:56.717323 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 10:32:56.717478 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":49}
Sep 29 10:32:56 dist-test-slave-9cj5 krb5kdc[11016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677976, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.62@KRBTEST.COM for kudu/127.20.92.60@KRBTEST.COM
I20260929 10:32:56.721690 20848 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:32:56.722582 11248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92fc0b3c06fa47adbbbbb65742e88150: Bootstrap starting.
I20260929 10:32:56.723634 11172 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40351 } attempt: 3
I20260929 10:32:56.723017 11071 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d3805215dcbe44cd808ee9537ae0227d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32978
I20260929 10:32:56.724949 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.715592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:40351 (local address 127.0.0.1:44886)
0929 10:32:56.715754 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:56.715768 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:56.715847 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:56.716454 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:56.716460 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:56.716650 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:56.716787 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:56.716793 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.717849 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.717853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:56.718634 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:56.718651 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.719322 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.719329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:56.719410 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:56.719861 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:32:56.719876 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:56.721251 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:56.723262 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:56.723266 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:56.723269 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:56.723481 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:56.723841 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:56.723845 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:56.723846 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:56.723918 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:56.724457 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:56.724460 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:56.724552 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 10:32:56.724616 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 10:32:56.724763 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 10:32:56.725612 11253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.715855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44886 (local address 127.20.92.60:40351)
0929 10:32:56.716213 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:56.716221 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:56.716234 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:32:56.716270 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:56.716274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:56.716319 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:56.716442 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:56.716892 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.717694 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.718755 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.719178 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.719653 (+ 475us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:56.721387 (+ 1734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:56.721398 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:56.721401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:56.721430 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:56.723116 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:56.723582 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:56.723588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:56.723590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:56.723673 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:56.724075 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:56.724078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:56.724079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:56.724317 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:56.725253 (+ 936us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:56.725324 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 10:32:56.725438 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":108,"threads_started":1}
I20260929 10:32:56.725679 11248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92fc0b3c06fa47adbbbbb65742e88150: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:56.726526 11248 log.cc:824] T 00000000000000000000000000000000 P 92fc0b3c06fa47adbbbbb65742e88150: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:56.728399 11097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3805215dcbe44cd808ee9537ae0227d: Bootstrap starting.
I20260929 10:32:56.728775 11248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92fc0b3c06fa47adbbbbb65742e88150: No bootstrap required, opened a new log
Sep 29 10:32:56 dist-test-slave-9cj5 krb5kdc[11016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677976, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.92.61@KRBTEST.COM for kudu/127.20.92.60@KRBTEST.COM
I20260929 10:32:56.730918 11097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3805215dcbe44cd808ee9537ae0227d: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:56.731820 11097 log.cc:824] T 00000000000000000000000000000000 P d3805215dcbe44cd808ee9537ae0227d: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:56.733139 11248 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92fc0b3c06fa47adbbbbb65742e88150 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3805215dcbe44cd808ee9537ae0227d" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45813 } } peers { permanent_uuid: "bbe3672e8af943d384ca88a908c02724" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 35155 } } peers { permanent_uuid: "92fc0b3c06fa47adbbbbb65742e88150" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40351 } }
I20260929 10:32:56.733373 11248 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92fc0b3c06fa47adbbbbb65742e88150 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92fc0b3c06fa47adbbbbb65742e88150, State: Initialized, Role: FOLLOWER
I20260929 10:32:56.733598 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.724605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.92.60:40351 (local address 127.0.0.1:44896)
0929 10:32:56.724746 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 10:32:56.724761 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:32:56.724830 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:32:56.725294 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:32:56.725297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:32:56.725484 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:32:56.725578 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:56.725587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.726547 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.726553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:56.727342 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:32:56.727353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.727911 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.727915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:32:56.728011 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:56.728622 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:32:56.728636 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:32:56.730112 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:32:56.732015 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:56.732018 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:56.732020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:56.732277 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:56.732575 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:32:56.732579 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:32:56.732582 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:32:56.732669 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:32:56.733086 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:32:56.733090 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:32:56.733201 (+ 111us) client_negotiation.cc:769] Sending connection context
0929 10:32:56.733310 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 10:32:56.733452 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 10:32:56.733628 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.724806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44896 (local address 127.20.92.60:40351)
0929 10:32:56.724897 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:56.724901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:56.724913 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:32:56.724946 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:56.724948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:56.724986 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:56.725060 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:56.725694 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.726429 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.727492 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.727794 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.727898 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:56.730255 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:56.730259 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:56.730262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:56.730284 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:56.731889 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:56.732387 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:56.732390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:56.732392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:56.732476 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:56.732783 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:56.732786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:56.732787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:56.732983 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:56.733085 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:56.733377 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 10:32:56.733491 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 10:32:56.733687 11248 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92fc0b3c06fa47adbbbbb65742e88150 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3805215dcbe44cd808ee9537ae0227d" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45813 } } peers { permanent_uuid: "bbe3672e8af943d384ca88a908c02724" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 35155 } } peers { permanent_uuid: "92fc0b3c06fa47adbbbbb65742e88150" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40351 } }
I20260929 10:32:56.733953 11097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3805215dcbe44cd808ee9537ae0227d: No bootstrap required, opened a new log
I20260929 10:32:56.735096 11248 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92fc0b3c06fa47adbbbbb65742e88150 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:32:56.734987 11256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92fc0b3c06fa47adbbbbb65742e88150 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3805215dcbe44cd808ee9537ae0227d" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45813 } } peers { permanent_uuid: "bbe3672e8af943d384ca88a908c02724" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 35155 } } peers { permanent_uuid: "92fc0b3c06fa47adbbbbb65742e88150" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40351 } } }
I20260929 10:32:56.735314 11256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92fc0b3c06fa47adbbbbb65742e88150 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:32:56.736584 11172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbe3672e8af943d384ca88a908c02724: Bootstrap starting.
I20260929 10:32:56.736841 11097 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3805215dcbe44cd808ee9537ae0227d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3805215dcbe44cd808ee9537ae0227d" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45813 } } peers { permanent_uuid: "bbe3672e8af943d384ca88a908c02724" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 35155 } } peers { permanent_uuid: "92fc0b3c06fa47adbbbbb65742e88150" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40351 } }
I20260929 10:32:56.737054 11097 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3805215dcbe44cd808ee9537ae0227d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3805215dcbe44cd808ee9537ae0227d, State: Initialized, Role: FOLLOWER
I20260929 10:32:56.737357 11097 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3805215dcbe44cd808ee9537ae0227d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3805215dcbe44cd808ee9537ae0227d" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45813 } } peers { permanent_uuid: "bbe3672e8af943d384ca88a908c02724" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 35155 } } peers { permanent_uuid: "92fc0b3c06fa47adbbbbb65742e88150" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40351 } }
I20260929 10:32:56.738710 11097 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3805215dcbe44cd808ee9537ae0227d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:32:56.738730 11071 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3805215dcbe44cd808ee9537ae0227d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 10:32:56.738839 11071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3805215dcbe44cd808ee9537ae0227d [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:56.739182 11172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbe3672e8af943d384ca88a908c02724: Neither blocks nor log segments found. Creating new log.
I20260929 10:32:56.740082 11172 log.cc:824] T 00000000000000000000000000000000 P bbe3672e8af943d384ca88a908c02724: Log is configured to *not* fsync() on all Append() calls
I20260929 10:32:56.740074 11071 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3805215dcbe44cd808ee9537ae0227d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3805215dcbe44cd808ee9537ae0227d" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45813 } } peers { permanent_uuid: "bbe3672e8af943d384ca88a908c02724" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 35155 } } peers { permanent_uuid: "92fc0b3c06fa47adbbbbb65742e88150" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40351 } }
I20260929 10:32:56.741480 11222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d3805215dcbe44cd808ee9537ae0227d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "92fc0b3c06fa47adbbbbb65742e88150"
I20260929 10:32:56.741706 11222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92fc0b3c06fa47adbbbbb65742e88150 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:56.739225 11179 master_runner.cc:186] Error getting master registration for 127.20.92.62:45813: OK, instance_id { permanent_uuid: "d3805215dcbe44cd808ee9537ae0227d" instance_seqno: 1790677976245297 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 10:32:56.742516 11179 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 10:32:56.742961 11222 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 92fc0b3c06fa47adbbbbb65742e88150 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3805215dcbe44cd808ee9537ae0227d in term 1.
W20260929 10:32:56.743690 11280 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d3805215dcbe44cd808ee9537ae0227d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:32:56.743762 11280 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:32:56.744421 11048 leader_election.cc:302] T 00000000000000000000000000000000 P d3805215dcbe44cd808ee9537ae0227d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92fc0b3c06fa47adbbbbb65742e88150, d3805215dcbe44cd808ee9537ae0227d; no voters:
I20260929 10:32:56.740957 11071 leader_election.cc:288] T 00000000000000000000000000000000 P d3805215dcbe44cd808ee9537ae0227d [CANDIDATE]: Term 1 election: Requested vote from peers bbe3672e8af943d384ca88a908c02724 (127.20.92.61:35155), 92fc0b3c06fa47adbbbbb65742e88150 (127.20.92.60:40351)
W20260929 10:32:56.743249 11050 tablet.cc:2369] T 00000000000000000000000000000000 P d3805215dcbe44cd808ee9537ae0227d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:32:56.748471 11172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbe3672e8af943d384ca88a908c02724: No bootstrap required, opened a new log
I20260929 10:32:56.745828 11029 master_runner.cc:186] Error getting master registration for 127.20.92.61:35155: OK, instance_id { permanent_uuid: "bbe3672e8af943d384ca88a908c02724" instance_seqno: 1790677976468357 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 10:32:56.749078 11029 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 10:32:56.749716 11267 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 92fc0b3c06fa47adbbbbb65742e88150: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:32:56.749876 11267 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:32:56.749920 11268 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3805215dcbe44cd808ee9537ae0227d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3805215dcbe44cd808ee9537ae0227d" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45813 } } peers { permanent_uuid: "bbe3672e8af943d384ca88a908c02724" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 35155 } } peers { permanent_uuid: "92fc0b3c06fa47adbbbbb65742e88150" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40351 } } }
I20260929 10:32:56.750061 11268 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3805215dcbe44cd808ee9537ae0227d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:32:56.750136 11268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3805215dcbe44cd808ee9537ae0227d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:32:56.750206 11268 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3805215dcbe44cd808ee9537ae0227d [term 1 LEADER]: Becoming Leader. State: Replica: d3805215dcbe44cd808ee9537ae0227d, State: Running, Role: LEADER
I20260929 10:32:56.750501 11268 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3805215dcbe44cd808ee9537ae0227d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3805215dcbe44cd808ee9537ae0227d" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45813 } } peers { permanent_uuid: "bbe3672e8af943d384ca88a908c02724" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 35155 } } peers { permanent_uuid: "92fc0b3c06fa47adbbbbb65742e88150" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40351 } }
I20260929 10:32:56.752568 11172 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbe3672e8af943d384ca88a908c02724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3805215dcbe44cd808ee9537ae0227d" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45813 } } peers { permanent_uuid: "bbe3672e8af943d384ca88a908c02724" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 35155 } } peers { permanent_uuid: "92fc0b3c06fa47adbbbbb65742e88150" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40351 } }
I20260929 10:32:56.752787 11172 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbe3672e8af943d384ca88a908c02724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe3672e8af943d384ca88a908c02724, State: Initialized, Role: FOLLOWER
I20260929 10:32:56.753093 11172 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbe3672e8af943d384ca88a908c02724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3805215dcbe44cd808ee9537ae0227d" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45813 } } peers { permanent_uuid: "bbe3672e8af943d384ca88a908c02724" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 35155 } } peers { permanent_uuid: "92fc0b3c06fa47adbbbbb65742e88150" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40351 } }
I20260929 10:32:56.753623 11290 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe3672e8af943d384ca88a908c02724 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3805215dcbe44cd808ee9537ae0227d" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45813 } } peers { permanent_uuid: "bbe3672e8af943d384ca88a908c02724" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 35155 } } peers { permanent_uuid: "92fc0b3c06fa47adbbbbb65742e88150" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40351 } } }
I20260929 10:32:56.753925 11290 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe3672e8af943d384ca88a908c02724 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:32:56.754498 11172 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbe3672e8af943d384ca88a908c02724 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:32:56.754782 11146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d3805215dcbe44cd808ee9537ae0227d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bbe3672e8af943d384ca88a908c02724"
I20260929 10:32:56.754988 11146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbe3672e8af943d384ca88a908c02724 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:32:56.756291 11146 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bbe3672e8af943d384ca88a908c02724 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3805215dcbe44cd808ee9537ae0227d in term 1.
W20260929 10:32:56.759807 11303 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bbe3672e8af943d384ca88a908c02724: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:32:56.759879 11303 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:32:56.762004 11283 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3805215dcbe44cd808ee9537ae0227d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3805215dcbe44cd808ee9537ae0227d. Latest consensus state: current_term: 1 leader_uuid: "d3805215dcbe44cd808ee9537ae0227d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3805215dcbe44cd808ee9537ae0227d" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45813 } } peers { permanent_uuid: "bbe3672e8af943d384ca88a908c02724" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 35155 } } peers { permanent_uuid: "92fc0b3c06fa47adbbbbb65742e88150" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40351 } } }
I20260929 10:32:56.762189 11283 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3805215dcbe44cd808ee9537ae0227d [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:56.774021 11305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:32:56.774624 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.752035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32994 (local address 127.20.92.62:45813)
0929 10:32:56.752167 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:56.752172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:56.754370 (+ 2198us) server_negotiation.cc:407] Connection header received
0929 10:32:56.754845 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:56.754849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:56.754900 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:56.754967 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:56.762191 (+ 7224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.763042 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.765180 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.765515 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.765596 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:56.768631 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:56.768635 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:56.768638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:56.768665 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:56.770351 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:56.773187 (+ 2836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:56.773191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:56.773193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:56.773275 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:56.773549 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:56.773553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:56.773555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:56.773794 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:56.773924 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:56.774343 (+ 419us) server_negotiation.cc:299] Negotiation successful
0929 10:32:56.774460 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 10:32:56.775061 11305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:32:56.780555 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.752483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44906 (local address 127.20.92.60:40351)
0929 10:32:56.752597 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:56.752601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:56.756536 (+ 3935us) server_negotiation.cc:407] Connection header received
0929 10:32:56.756600 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:56.756605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:56.756658 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:56.756742 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:56.763341 (+ 6599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.764106 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.765260 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.765596 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.765664 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:56.770815 (+ 5151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:56.770819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:56.770822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:56.770847 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:56.772474 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:56.776964 (+ 4490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:56.776967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:56.776970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:56.777062 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:56.777386 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:56.777391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:56.777396 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:56.777619 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:56.777726 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:56.780281 (+ 2555us) server_negotiation.cc:299] Negotiation successful
0929 10:32:56.780400 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 10:32:56.782622 11178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:56.751632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47098 (local address 127.20.92.61:35155)
0929 10:32:56.751764 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:56.751769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:56.762010 (+ 10241us) server_negotiation.cc:407] Connection header received
0929 10:32:56.764266 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:56.764270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:56.764320 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:56.764370 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:56.766688 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.767442 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.768534 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:56.768874 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:56.769224 (+ 350us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:56.778050 (+ 8826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:32:56.778054 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:32:56.778057 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:32:56.778083 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:32:56.779738 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:56.781255 (+ 1517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:56.781258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:56.781260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:56.781346 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:32:56.781647 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:32:56.781651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:32:56.781652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:32:56.781890 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:32:56.782015 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:56.782344 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 10:32:56.782459 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 10:32:56.796190 11222 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 92fc0b3c06fa47adbbbbb65742e88150 [term 1 FOLLOWER]: Refusing update from remote peer d3805215dcbe44cd808ee9537ae0227d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:32:56.796866 11283 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3805215dcbe44cd808ee9537ae0227d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92fc0b3c06fa47adbbbbb65742e88150" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:32:56.804069 11256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92fc0b3c06fa47adbbbbb65742e88150 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3805215dcbe44cd808ee9537ae0227d. Latest consensus state: current_term: 1 leader_uuid: "d3805215dcbe44cd808ee9537ae0227d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3805215dcbe44cd808ee9537ae0227d" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45813 } } peers { permanent_uuid: "bbe3672e8af943d384ca88a908c02724" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 35155 } } peers { permanent_uuid: "92fc0b3c06fa47adbbbbb65742e88150" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40351 } } }
I20260929 10:32:56.804196 11256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92fc0b3c06fa47adbbbbb65742e88150 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:32:56.805101 11146 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bbe3672e8af943d384ca88a908c02724 [term 1 FOLLOWER]: Refusing update from remote peer d3805215dcbe44cd808ee9537ae0227d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:32:56.805626 11283 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3805215dcbe44cd808ee9537ae0227d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbe3672e8af943d384ca88a908c02724" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 35155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:32:56.809759 11290 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe3672e8af943d384ca88a908c02724 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3805215dcbe44cd808ee9537ae0227d. Latest consensus state: current_term: 1 leader_uuid: "d3805215dcbe44cd808ee9537ae0227d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3805215dcbe44cd808ee9537ae0227d" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45813 } } peers { permanent_uuid: "bbe3672e8af943d384ca88a908c02724" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 35155 } } peers { permanent_uuid: "92fc0b3c06fa47adbbbbb65742e88150" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40351 } } }
I20260929 10:32:56.809877 11290 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe3672e8af943d384ca88a908c02724 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:32:56.816102 11283 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3805215dcbe44cd808ee9537ae0227d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d3805215dcbe44cd808ee9537ae0227d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3805215dcbe44cd808ee9537ae0227d" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45813 } } peers { permanent_uuid: "bbe3672e8af943d384ca88a908c02724" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 35155 } } peers { permanent_uuid: "92fc0b3c06fa47adbbbbb65742e88150" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40351 } } }
I20260929 10:32:56.816233 11283 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3805215dcbe44cd808ee9537ae0227d [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:56.817365 11290 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe3672e8af943d384ca88a908c02724 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d3805215dcbe44cd808ee9537ae0227d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3805215dcbe44cd808ee9537ae0227d" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45813 } } peers { permanent_uuid: "bbe3672e8af943d384ca88a908c02724" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 35155 } } peers { permanent_uuid: "92fc0b3c06fa47adbbbbb65742e88150" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40351 } } }
I20260929 10:32:56.817477 11290 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe3672e8af943d384ca88a908c02724 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:32:56.817848 11306 mvcc.cc:205] Tried to move back new op lower bound from 7334616992949211136 to 7334616992778874880. Current Snapshot: MvccSnapshot[applied={T|T < 7334616992949211136}]
I20260929 10:32:56.819759 11309 mvcc.cc:205] Tried to move back new op lower bound from 7334616992949211136 to 7334616992778874880. Current Snapshot: MvccSnapshot[applied={T|T < 7334616992949211136}]
I20260929 10:32:56.823881 11268 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3805215dcbe44cd808ee9537ae0227d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d3805215dcbe44cd808ee9537ae0227d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3805215dcbe44cd808ee9537ae0227d" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45813 } } peers { permanent_uuid: "bbe3672e8af943d384ca88a908c02724" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 35155 } } peers { permanent_uuid: "92fc0b3c06fa47adbbbbb65742e88150" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40351 } } }
I20260929 10:32:56.824031 11268 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3805215dcbe44cd808ee9537ae0227d [sys.catalog]: This master's current role is: LEADER
I20260929 10:32:56.825054 11256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92fc0b3c06fa47adbbbbb65742e88150 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d3805215dcbe44cd808ee9537ae0227d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3805215dcbe44cd808ee9537ae0227d" member_type: VOTER last_known_addr { host: "127.20.92.62" port: 45813 } } peers { permanent_uuid: "bbe3672e8af943d384ca88a908c02724" member_type: VOTER last_known_addr { host: "127.20.92.61" port: 35155 } } peers { permanent_uuid: "92fc0b3c06fa47adbbbbb65742e88150" member_type: VOTER last_known_addr { host: "127.20.92.60" port: 40351 } } }
I20260929 10:32:56.825163 11256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92fc0b3c06fa47adbbbbb65742e88150 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:32:56.825618 11307 mvcc.cc:205] Tried to move back new op lower bound from 7334616992949211136 to 7334616992778874880. Current Snapshot: MvccSnapshot[applied={T|T < 7334616992949211136}]
I20260929 10:32:56.829100 11305 catalog_manager.cc:1347] Generated new cluster ID: 7ea9ce8a37f64829a3aa849e3c3f3dc1
I20260929 10:32:56.829175 11305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:32:56.854053 11305 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:32:56.855381 11305 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:32:56.881577 11305 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3805215dcbe44cd808ee9537ae0227d: Generated new TSK 0
I20260929 10:32:56.882530 11305 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 10:32:56.908195 11251 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:32:56.905765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44922 (local address 127.20.92.60:40351)
0929 10:32:56.905910 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:56.905915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:56.907092 (+ 1177us) server_negotiation.cc:407] Connection header received
0929 10:32:56.907315 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:56.907319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:56.907369 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:56.907422 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:56.908058 (+ 636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44922: BlockingRecv error: recv got EOF from 127.0.0.1:44922 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 10:32:56.909706 11100 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:32:56.906450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33002 (local address 127.20.92.62:45813)
0929 10:32:56.906574 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:56.906580 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:56.908686 (+ 2106us) server_negotiation.cc:407] Connection header received
0929 10:32:56.908941 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:56.908945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:56.908998 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:56.909051 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:56.909576 (+ 525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33002: BlockingRecv error: recv got EOF from 127.0.0.1:33002 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 10:32:56.911581 11178 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:32:56.910321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47102 (local address 127.20.92.61:35155)
0929 10:32:56.910479 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:56.910486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:56.910709 (+ 223us) server_negotiation.cc:407] Connection header received
0929 10:32:56.910884 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:56.910887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:56.910933 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:56.910985 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:56.911471 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47102: BlockingRecv error: recv got EOF from 127.0.0.1:47102 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 10:32:56.915918 11178 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:32:56.914420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47112 (local address 127.20.92.61:35155)
0929 10:32:56.914518 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:56.914523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:56.914943 (+ 420us) server_negotiation.cc:407] Connection header received
0929 10:32:56.915155 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:56.915158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:56.915231 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:56.915286 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:56.915815 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47112: BlockingRecv error: recv got EOF from 127.0.0.1:47112 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260929 10:32:56.920974 11178 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:32:56.919496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47126 (local address 127.20.92.61:35155)
0929 10:32:56.919596 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:56.919600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:56.920080 (+ 480us) server_negotiation.cc:407] Connection header received
0929 10:32:56.920275 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:56.920278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:56.920327 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:56.920377 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:56.920877 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47126: BlockingRecv error: recv got EOF from 127.0.0.1:47126 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 10:32:56.926856 11178 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:32:56.925272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47136 (local address 127.20.92.61:35155)
0929 10:32:56.925371 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:56.925375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:56.925887 (+ 512us) server_negotiation.cc:407] Connection header received
0929 10:32:56.926105 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:56.926116 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:56.926176 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:56.926232 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:56.926756 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47136: BlockingRecv error: recv got EOF from 127.0.0.1:47136 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260929 10:32:56.934865 11178 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:32:56.933395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47148 (local address 127.20.92.61:35155)
0929 10:32:56.933574 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:56.933579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:56.933878 (+ 299us) server_negotiation.cc:407] Connection header received
0929 10:32:56.934087 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:56.934090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:56.934144 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:56.934201 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:56.934756 (+ 555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47148: BlockingRecv error: recv got EOF from 127.0.0.1:47148 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
W20260929 10:32:56.946897 11178 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:32:56.945245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47158 (local address 127.20.92.61:35155)
0929 10:32:56.945377 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:56.945381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:56.945956 (+ 575us) server_negotiation.cc:407] Connection header received
0929 10:32:56.946148 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:56.946151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:56.946205 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:56.946255 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:56.946783 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47158: BlockingRecv error: recv got EOF from 127.0.0.1:47158 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 10:32:56.966487 11178 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:32:56.965188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47166 (local address 127.20.92.61:35155)
0929 10:32:56.965309 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:56.965313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:56.965567 (+ 254us) server_negotiation.cc:407] Connection header received
0929 10:32:56.965763 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:56.965768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:56.965827 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:56.965918 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:56.966374 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47166: BlockingRecv error: recv got EOF from 127.0.0.1:47166 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 10:32:57.001869 11178 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:32:57.000683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47178 (local address 127.20.92.61:35155)
0929 10:32:57.000882 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:57.000886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:57.000995 (+ 109us) server_negotiation.cc:407] Connection header received
0929 10:32:57.001133 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:57.001136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:57.001184 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:57.001258 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:57.001704 (+ 446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47178: BlockingRecv error: recv got EOF from 127.0.0.1:47178 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
W20260929 10:32:57.069103 11178 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:32:57.067953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47192 (local address 127.20.92.61:35155)
0929 10:32:57.068108 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:57.068115 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:57.068253 (+ 138us) server_negotiation.cc:407] Connection header received
0929 10:32:57.068374 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:57.068376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:57.068425 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:57.068498 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:57.068953 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47192: BlockingRecv error: recv got EOF from 127.0.0.1:47192 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 10:32:57.200513 11178 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:32:57.199280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47194 (local address 127.20.92.61:35155)
0929 10:32:57.199422 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:57.199426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:57.199625 (+ 199us) server_negotiation.cc:407] Connection header received
0929 10:32:57.199808 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:57.199811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:57.199859 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:57.199950 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:57.200409 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47194: BlockingRecv error: recv got EOF from 127.0.0.1:47194 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 10:32:57.460320 11178 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:32:57.459083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47202 (local address 127.20.92.61:35155)
0929 10:32:57.459226 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:57.459232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:57.459325 (+ 93us) server_negotiation.cc:407] Connection header received
0929 10:32:57.459547 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:57.459550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:57.459600 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:57.459702 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:32:57.460202 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47202: BlockingRecv error: recv got EOF from 127.0.0.1:47202 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 10:32:57.751199 11267 catalog_manager.cc:1259] Loaded cluster ID: 7ea9ce8a37f64829a3aa849e3c3f3dc1
I20260929 10:32:57.751286 11267 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 92fc0b3c06fa47adbbbbb65742e88150: loading cluster ID for follower catalog manager: success
I20260929 10:32:57.752985 11267 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 92fc0b3c06fa47adbbbbb65742e88150: acquiring CA information for follower catalog manager: success
I20260929 10:32:57.754298 11267 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 92fc0b3c06fa47adbbbbb65742e88150: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 10:32:57.760934 11303 catalog_manager.cc:1259] Loaded cluster ID: 7ea9ce8a37f64829a3aa849e3c3f3dc1
I20260929 10:32:57.761052 11303 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bbe3672e8af943d384ca88a908c02724: loading cluster ID for follower catalog manager: success
I20260929 10:32:57.762677 11303 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bbe3672e8af943d384ca88a908c02724: acquiring CA information for follower catalog manager: success
I20260929 10:32:57.763772 11303 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bbe3672e8af943d384ca88a908c02724: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 10:32:57.978633 11377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:57.974646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47208 (local address 127.20.92.61:35155)
0929 10:32:57.974900 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:57.974907 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:57.974926 (+ 19us) server_negotiation.cc:407] Connection header received
0929 10:32:57.975116 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:57.975120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:57.975253 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:57.975371 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:32:57.975671 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:57.976195 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:57.977056 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:57.977287 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:57.977399 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:57.977730 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:32:57.978215 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:32:57.978352 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:57.978414 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 10:32:57.978467 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":70,"threads_started":1}
I20260929 10:32:57.984905 11383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:57.981548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44936 (local address 127.20.92.60:40351)
0929 10:32:57.981825 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:57.981828 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:57.981845 (+ 17us) server_negotiation.cc:407] Connection header received
0929 10:32:57.981916 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:57.981919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:57.982019 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:57.982123 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:32:57.982320 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:57.982846 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:57.983618 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:57.983821 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:57.983899 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:57.984138 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:32:57.984508 (+ 370us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:32:57.984597 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:57.984664 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 10:32:57.984708 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":74,"threads_started":1}
I20260929 10:32:57.991952 11377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:57.987830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47222 (local address 127.20.92.61:35155)
0929 10:32:57.987950 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:57.987954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:57.988175 (+ 221us) server_negotiation.cc:407] Connection header received
0929 10:32:57.988383 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:57.988389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:57.988451 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:57.988547 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:32:57.988803 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:57.989595 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:57.990770 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:57.990976 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:57.991090 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:57.991439 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:32:57.991525 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:32:57.991624 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:57.991740 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 10:32:57.991794 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 10:32:57.991986 11383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:57.987836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44948 (local address 127.20.92.60:40351)
0929 10:32:57.988046 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:57.988050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:57.988302 (+ 252us) server_negotiation.cc:407] Connection header received
0929 10:32:57.988431 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:57.988434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:57.988482 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:57.988546 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:32:57.988922 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:57.989406 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:57.990707 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:57.991031 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:57.991128 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:57.991513 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:32:57.991591 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:32:57.991668 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:57.991807 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 10:32:57.991857 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260929 10:32:57.992441 11391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:32:57.987968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33018 (local address 127.20.92.62:45813)
0929 10:32:57.988180 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 10:32:57.988184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:32:57.988252 (+ 68us) server_negotiation.cc:407] Connection header received
0929 10:32:57.988456 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:32:57.988460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:32:57.988609 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:32:57.988731 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:32:57.988979 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:57.989789 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:57.990657 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:32:57.991001 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:32:57.991110 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:32:57.991597 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:32:57.991934 (+ 337us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:32:57.992019 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 10:32:57.992237 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 10:32:57.992304 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":71,"threads_started":1}
I20260929 10:32:57.995603 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 11029
I20260929 10:32:58.002192 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 11104
I20260929 10:32:58.008147 20848 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Q3Hzv/build/debug/bin/kudu with pid 11179
2026-09-29T10:32:58Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3111 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6249 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (298714 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST